[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:22:58 dist-test-slave-q5g8 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:22:58 dist-test-slave-q5g8 krb5kdc[395](info): set up 2 sockets
Sep 26 23:22:58 dist-test-slave-q5g8 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 23:23:00.013583 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:00 dist-test-slave-q5g8 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:00.210789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36395
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:00.589071 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:00.589403 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:00.589567 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:00.599022 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:00.599197 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:00.599303 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:00.599426 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:00.599494 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:00.610208 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:00.611819 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:00.613548 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:00.628430 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:00.628350 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:00.628350 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:00.629479 408 server_base.cc:1034] running on GCE node
I20260926 23:23:00.630415 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:00.632193 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:00.633486 408 hybrid_clock.cc:648] HybridClock initialized: now 1790464980633389 us; error 88 us; skew 500 ppm
I20260926 23:23:00.633949 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:00 dist-test-slave-q5g8 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:00.637663 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:00.639454 408 webserver.cc:460] Webserver started at http://127.0.97.62:40187/ using document root <none> and password file <none>
I20260926 23:23:00.640223 408 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:00.640347 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:00.640621 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:00.644088 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "b8870c8665324aacadc851764e4b231b"
format_stamp: "Formatted at 2026-09-26 23:23:00 on dist-test-slave-q5g8"
server_key: "25d65bc423ad81e2d1da6ac495d7a3c5"
server_key_iv: "2b6efc119d83d586f759cd1014e9edf1"
server_key_version: "encryptionkey@0"
I20260926 23:23:00.645020 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "b8870c8665324aacadc851764e4b231b"
format_stamp: "Formatted at 2026-09-26 23:23:00 on dist-test-slave-q5g8"
server_key: "25d65bc423ad81e2d1da6ac495d7a3c5"
server_key_iv: "2b6efc119d83d586f759cd1014e9edf1"
server_key_version: "encryptionkey@0"
I20260926 23:23:00.651010 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:23:00.655236 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:00.656947 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:00.657161 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "b8870c8665324aacadc851764e4b231b"
format_stamp: "Formatted at 2026-09-26 23:23:00 on dist-test-slave-q5g8"
server_key: "25d65bc423ad81e2d1da6ac495d7a3c5"
server_key_iv: "2b6efc119d83d586f759cd1014e9edf1"
server_key_version: "encryptionkey@0"
I20260926 23:23:00.657402 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:00.684476 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:00.689445 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:00.689774 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:00.712702 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36395
I20260926 23:23:00.712775 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36395 every 8 connection(s)
I20260926 23:23:00.714275 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:00.718321 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:00.723785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 408
I20260926 23:23:00.724151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:00.726958 388 external_mini_cluster.cc:1428] Setting key 0ffc71ee0987abc8fbf040eebffd89ef
I20260926 23:23:00.733336 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b: Bootstrap starting.
I20260926 23:23:00.737068 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:00.738113 475 log.cc:824] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:00.741094 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b: No bootstrap required, opened a new log
I20260926 23:23:00.751906 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8870c8665324aacadc851764e4b231b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36395 } }
I20260926 23:23:00.752350 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:00.752472 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8870c8665324aacadc851764e4b231b, State: Initialized, Role: FOLLOWER
I20260926 23:23:00.753286 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8870c8665324aacadc851764e4b231b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36395 } }
I20260926 23:23:00.753614 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:00.753786 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:00.753985 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:00.757529 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8870c8665324aacadc851764e4b231b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36395 } }
I20260926 23:23:00.758307 475 leader_election.cc:302] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8870c8665324aacadc851764e4b231b; no voters:
I20260926 23:23:00.759585 475 leader_election.cc:288] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:00.759678 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 23:23:00 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:00.761438 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [term 1 LEADER]: Becoming Leader. State: Replica: b8870c8665324aacadc851764e4b231b, State: Running, Role: LEADER
I20260926 23:23:00.762337 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8870c8665324aacadc851764e4b231b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36395 } }
I20260926 23:23:00.763935 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:00.766201 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8870c8665324aacadc851764e4b231b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8870c8665324aacadc851764e4b231b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36395 } } }
I20260926 23:23:00.766697 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:00.767246 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8870c8665324aacadc851764e4b231b. Latest consensus state: current_term: 1 leader_uuid: "b8870c8665324aacadc851764e4b231b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8870c8665324aacadc851764e4b231b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36395 } } }
I20260926 23:23:00.767611 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:00.771795 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:00.778221 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:00.732291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.0.97.62:36395)
0926 23:23:00.733284 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:00.733302 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:00.734706 (+ 1404us) server_negotiation.cc:407] Connection header received
0926 23:23:00.743782 (+ 9076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:00.743814 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:00.744406 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:00.745498 (+ 1092us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:00.748369 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:00.750601 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:00.752599 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:00.753390 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:00.753603 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:00.763366 (+ 9763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:00.763381 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:00.763406 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:00.763478 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:00.769390 (+ 5912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:00.771080 (+ 1690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:00.771096 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:00.771110 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:00.771502 (+ 392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:00.772249 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:00.772254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:00.772258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:00.773121 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:00.775602 (+ 2481us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:00.775984 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 23:23:00.776551 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":466,"threads_started":1}
I20260926 23:23:00.782996 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:00.792513 485 catalog_manager.cc:1347] Generated new cluster ID: 7b24cf282bb04c4d9644b43b895d6731
I20260926 23:23:00.792658 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:00.810681 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:00.812111 485 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:00.828037 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8870c8665324aacadc851764e4b231b: Generated new TSK 0
I20260926 23:23:00.829083 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:00.885353 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36395
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:01.281826 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:01.282141 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:01.282317 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:01.290179 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:01.290347 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:01.290557 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:01.301726 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:01.303225 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:01.304801 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:01.319654 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:01.319972 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:01.320263 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:01.321494 501 server_base.cc:1034] running on GCE node
I20260926 23:23:01.322292 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:01.323402 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:01.324663 501 hybrid_clock.cc:648] HybridClock initialized: now 1790464981324604 us; error 58 us; skew 500 ppm
I20260926 23:23:01.325112 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:01 dist-test-slave-q5g8 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:01.328826 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:01.330466 501 webserver.cc:460] Webserver started at http://127.0.97.1:35335/ using document root <none> and password file <none>
I20260926 23:23:01.331223 501 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:01.331343 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:01.331612 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:01.334868 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "72bcb4fcca3f419c92862b2ea465dfce"
format_stamp: "Formatted at 2026-09-26 23:23:01 on dist-test-slave-q5g8"
server_key: "f72e8ad0c3ad65a6af1c82990ba80c29"
server_key_iv: "ad7918af3017be10f1c5884cb80e143d"
server_key_version: "encryptionkey@0"
I20260926 23:23:01.335690 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "72bcb4fcca3f419c92862b2ea465dfce"
format_stamp: "Formatted at 2026-09-26 23:23:01 on dist-test-slave-q5g8"
server_key: "f72e8ad0c3ad65a6af1c82990ba80c29"
server_key_iv: "ad7918af3017be10f1c5884cb80e143d"
server_key_version: "encryptionkey@0"
I20260926 23:23:01.341780 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:23:01.346192 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:01.347918 501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:23:01.348102 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "72bcb4fcca3f419c92862b2ea465dfce"
format_stamp: "Formatted at 2026-09-26 23:23:01 on dist-test-slave-q5g8"
server_key: "f72e8ad0c3ad65a6af1c82990ba80c29"
server_key_iv: "ad7918af3017be10f1c5884cb80e143d"
server_key_version: "encryptionkey@0"
I20260926 23:23:01.348332 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:01.363255 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:01.368027 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:01.368297 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:01.369416 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:01.370855 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:01.370935 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:01.371023 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:01.371068 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:01.409164 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36939
I20260926 23:23:01.409276 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36939 every 8 connection(s)
I20260926 23:23:01.410498 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:01.418054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 501
I20260926 23:23:01.418298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:01.418900 388 external_mini_cluster.cc:1428] Setting key dd04a0fae9874f8c8536a8b321822603
Sep 26 23:23:01 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:01.433483 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:01.413619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43599 (local address 127.0.97.62:36395)
0926 23:23:01.414094 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:01.414099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:01.414989 (+ 890us) server_negotiation.cc:407] Connection header received
0926 23:23:01.416973 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:01.416979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:01.417074 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:01.417194 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:01.418906 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.419832 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.421652 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.422285 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.422425 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:01.426246 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:01.426251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:01.426255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:01.426292 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:01.429652 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:01.430618 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:01.430624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:01.430627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:01.430765 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:01.431329 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:01.431334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:01.431336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:01.431665 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:01.431869 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:01.432777 (+ 908us) server_negotiation.cc:299] Negotiation successful
0926 23:23:01.433052 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260926 23:23:01.434698 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:01.414099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36395 (local address 127.0.97.1:43599)
0926 23:23:01.414804 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:01.414847 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:01.416598 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:01.417452 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:01.417462 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:01.418003 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:01.418717 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:01.418734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.420010 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.420015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:01.421499 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:01.421513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.422540 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.422547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:01.422788 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:01.423539 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:01.423563 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:01.425946 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:01.429863 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:01.429875 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:01.429901 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:01.430445 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:01.430898 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:01.430904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:01.430908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:01.431160 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:01.431843 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:01.431853 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:01.432289 (+ 436us) client_negotiation.cc:769] Sending connection context
0926 23:23:01.432608 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 23:23:01.433334 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":273,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:23:01.436753 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36395
I20260926 23:23:01.437095 629 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:01.437922 629 heartbeater.cc:511] Master 127.0.97.62:36395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:23:01.440277 440 ts_manager.cc:194] Registered new tserver with Master: 72bcb4fcca3f419c92862b2ea465dfce (127.0.97.1:36939)
I20260926 23:23:01.441901 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:01.461170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36395
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:01.831163 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:01.831483 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:01.831583 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:01.839509 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:01.839632 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:01.839749 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:01.849879 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:01.851209 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:01.852556 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:01.867587 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:01.867587 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:01.867892 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:01.868422 635 server_base.cc:1034] running on GCE node
I20260926 23:23:01.869158 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:01.870168 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:01.871419 635 hybrid_clock.cc:648] HybridClock initialized: now 1790464981871362 us; error 55 us; skew 500 ppm
I20260926 23:23:01.871776 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:01 dist-test-slave-q5g8 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:01.875788 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:01.877668 635 webserver.cc:460] Webserver started at http://127.0.97.2:42981/ using document root <none> and password file <none>
I20260926 23:23:01.878594 635 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:01.878719 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:01.879055 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:01.883307 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "66912c34513e4dacb86783cadca118bb"
format_stamp: "Formatted at 2026-09-26 23:23:01 on dist-test-slave-q5g8"
server_key: "05e592f7baf114fc516cfb0cd9a23f1d"
server_key_iv: "a5b6d6024704125f1e8fe72ac9ef2ad7"
server_key_version: "encryptionkey@0"
I20260926 23:23:01.884192 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "66912c34513e4dacb86783cadca118bb"
format_stamp: "Formatted at 2026-09-26 23:23:01 on dist-test-slave-q5g8"
server_key: "05e592f7baf114fc516cfb0cd9a23f1d"
server_key_iv: "a5b6d6024704125f1e8fe72ac9ef2ad7"
server_key_version: "encryptionkey@0"
I20260926 23:23:01.890071 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 23:23:01.893972 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:01.895507 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:23:01.895720 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "66912c34513e4dacb86783cadca118bb"
format_stamp: "Formatted at 2026-09-26 23:23:01 on dist-test-slave-q5g8"
server_key: "05e592f7baf114fc516cfb0cd9a23f1d"
server_key_iv: "a5b6d6024704125f1e8fe72ac9ef2ad7"
server_key_version: "encryptionkey@0"
I20260926 23:23:01.895959 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:01.928423 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:01.934195 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:01.934530 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:01.935658 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:01.937268 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:01.937345 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:01.937441 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:01.937487 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:01.980796 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43057
I20260926 23:23:01.980834 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43057 every 8 connection(s)
I20260926 23:23:01.982254 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:01.990551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 635
I20260926 23:23:01.990777 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:01.991925 388 external_mini_cluster.cc:1428] Setting key 2fcfb8dd90db3ed67b46d126f3881537
Sep 26 23:23:01 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:02.008313 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:01.985180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56083 (local address 127.0.97.62:36395)
0926 23:23:01.985681 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:01.985686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:01.986843 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 23:23:01.988805 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:01.988810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:01.988883 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:01.988987 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:01.990971 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.992572 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.994984 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.995862 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.996054 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.000129 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:02.000135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:02.000139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:02.000179 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:02.004188 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.005230 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.005239 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.005246 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.005424 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.005945 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.005951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.005956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.006394 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:02.006577 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:02.007453 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 23:23:02.007750 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
I20260926 23:23:02.009073 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:01.985861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36395 (local address 127.0.97.2:56083)
0926 23:23:01.986597 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:01.986695 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:01.988448 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:01.989310 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:01.989320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:01.990008 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:01.990657 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:01.990682 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.992824 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.992830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:01.994803 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:01.994818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:01.996138 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:01.996144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:01.996385 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:01.997242 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:01.997274 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:01.999845 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:02.004419 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:02.004428 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:02.004445 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:02.005071 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:02.005561 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:02.005565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:02.005567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:02.005799 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:02.006530 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:02.006537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:02.007023 (+ 486us) client_negotiation.cc:769] Sending connection context
0926 23:23:02.007281 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 23:23:02.007736 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":310,"threads_started":1}
I20260926 23:23:02.011014 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36395
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:23:02.011404 763 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:02.012241 763 heartbeater.cc:511] Master 127.0.97.62:36395 requested a full tablet report, sending...
I20260926 23:23:02.014156 440 ts_manager.cc:194] Registered new tserver with Master: 66912c34513e4dacb86783cadca118bb (127.0.97.2:43057)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:23:02.015419 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:02.041633 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36395
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:02.414255 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:02.414588 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:02.414763 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:02.422612 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:02.422809 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:02.422998 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:02.434297 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:02.435793 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:02.437748 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:23:02.445081 629 heartbeater.cc:503] Master 127.0.97.62:36395 was elected leader, sending a full tablet report...
W20260926 23:23:02.453730 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:02.453768 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:02.453627 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:02.454880 769 server_base.cc:1034] running on GCE node
I20260926 23:23:02.455700 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:02.456997 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:02.458249 769 hybrid_clock.cc:648] HybridClock initialized: now 1790464982458186 us; error 57 us; skew 500 ppm
I20260926 23:23:02.458619 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:02.462723 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:02.464397 769 webserver.cc:460] Webserver started at http://127.0.97.3:36855/ using document root <none> and password file <none>
I20260926 23:23:02.465317 769 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:02.465459 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:02.465850 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:02.469167 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "24bbb8324a57450087bf625c98b40aff"
format_stamp: "Formatted at 2026-09-26 23:23:02 on dist-test-slave-q5g8"
server_key: "7b3cdf7d10e45ddc1ea9298319cb539f"
server_key_iv: "46e987540be8fae59b676028c8eb8c91"
server_key_version: "encryptionkey@0"
I20260926 23:23:02.470100 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "24bbb8324a57450087bf625c98b40aff"
format_stamp: "Formatted at 2026-09-26 23:23:02 on dist-test-slave-q5g8"
server_key: "7b3cdf7d10e45ddc1ea9298319cb539f"
server_key_iv: "46e987540be8fae59b676028c8eb8c91"
server_key_version: "encryptionkey@0"
I20260926 23:23:02.476105 769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260926 23:23:02.480223 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:02.481839 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:02.482035 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "24bbb8324a57450087bf625c98b40aff"
format_stamp: "Formatted at 2026-09-26 23:23:02 on dist-test-slave-q5g8"
server_key: "7b3cdf7d10e45ddc1ea9298319cb539f"
server_key_iv: "46e987540be8fae59b676028c8eb8c91"
server_key_version: "encryptionkey@0"
I20260926 23:23:02.482280 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:02.504475 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:02.509063 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:02.509339 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:02.510483 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:02.511868 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:02.511963 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:02.512051 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:02.512095 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:02.550653 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38557
I20260926 23:23:02.550709 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38557 every 8 connection(s)
I20260926 23:23:02.552008 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:02.561234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 769
I20260926 23:23:02.561424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:02.561865 388 external_mini_cluster.cc:1428] Setting key 5116f5573ace77f6348303a933e179b5
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:02.575737 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:02.555531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57471 (local address 127.0.97.62:36395)
0926 23:23:02.555977 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:02.555982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:02.557128 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 23:23:02.559272 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:02.559278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:02.559405 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:02.559546 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:02.561512 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.562517 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.564641 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.565160 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.565308 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.569489 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:02.569495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:02.569498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:02.569545 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:02.572535 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.573522 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.573526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.573528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.573650 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.574072 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.574076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.574078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.574353 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:02.574464 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:02.575259 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 23:23:02.575434 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":210,"threads_started":1}
I20260926 23:23:02.576915 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:02.556141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36395 (local address 127.0.97.3:57471)
0926 23:23:02.556944 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:02.556990 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:02.558908 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:02.559829 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:02.559838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:02.560433 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:02.561267 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:02.561288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.562714 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.562718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:02.564492 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:02.564509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.565419 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.565431 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:02.565811 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.566563 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:02.566595 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:02.569250 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:02.572742 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:02.572751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:02.572767 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:02.573391 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:02.573759 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:02.573762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:02.573764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:02.573962 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:02.574497 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:02.574504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:02.574878 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 23:23:02.575141 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 23:23:02.575564 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260926 23:23:02.578900 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36395
I20260926 23:23:02.579233 897 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:02.579963 897 heartbeater.cc:511] Master 127.0.97.62:36395 requested a full tablet report, sending...
I20260926 23:23:02.581499 440 ts_manager.cc:194] Registered new tserver with Master: 24bbb8324a57450087bf625c98b40aff (127.0.97.3:38557)
I20260926 23:23:02.582314 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57471
I20260926 23:23:02.591164 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:02.642715 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:02.622313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.0.97.1:36939)
0926 23:23:02.622967 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:02.622977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:02.623012 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:23:02.623194 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:02.623202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:02.623421 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:02.623592 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:02.625156 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.626138 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.627412 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.627982 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.628121 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.637109 (+ 8988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:02.637126 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:02.637142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:02.637193 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:02.639985 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.640739 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.640746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.640749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.640872 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.641297 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.641301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.641304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.641690 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:02.641922 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:02.642143 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 23:23:02.642352 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":323,"threads_started":1}
W20260926 23:23:02.643945 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39224
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:02.690207 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:02.674930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.0.97.1:36939)
0926 23:23:02.675152 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:02.675161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:02.675563 (+ 402us) server_negotiation.cc:407] Connection header received
0926 23:23:02.675796 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:02.675802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:02.675874 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:02.676005 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:02.677506 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.678224 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.679553 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.679997 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.680142 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.684789 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:02.684795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:02.684798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:02.684849 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:02.687490 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.688202 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.688209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.688213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.688361 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.688808 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.688815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.688820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.689083 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:02.689344 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:02.689655 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 23:23:02.689882 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:23:02.691756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 501
I20260926 23:23:02.704331 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:02.692669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.0.97.1:36939)
0926 23:23:02.692861 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:02.692865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:02.693264 (+ 399us) server_negotiation.cc:407] Connection header received
0926 23:23:02.693370 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:02.693373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:02.693426 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:02.693506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:02.694673 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.695407 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.696625 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.697190 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.697296 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.699485 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:02.699495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:02.699498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:02.699530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:02.702217 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.702826 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.702830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.702832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.702950 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.703233 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.703236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.703238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.703480 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:02.703648 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:02.703899 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 23:23:02.704050 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:23:02.705080 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:02.796957 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 501
I20260926 23:23:02.814695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 635
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:02.830646 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:02.815631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.0.97.2:43057)
0926 23:23:02.816220 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:02.816228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:02.816258 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:23:02.816325 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:02.816332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:02.816562 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:02.816789 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:02.817987 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.819059 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.820306 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.821041 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.821143 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.825249 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:02.825262 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:02.825278 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:02.825314 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:02.827978 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.828636 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.828644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.828646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.828940 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.829299 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.829302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.829304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.829639 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:02.829873 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:02.830135 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 23:23:02.830340 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":302,"threads_started":1}
I20260926 23:23:02.831902 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:02.915979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 635
I20260926 23:23:02.939534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 769
Sep 26 23:23:02 dist-test-slave-q5g8 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:02.957131 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:02.940739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.0.97.3:38557)
0926 23:23:02.941379 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:02.941386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:02.941415 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:23:02.941474 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:02.941481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:02.941693 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:02.941835 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:02.943006 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.944027 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.945383 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:02.945991 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:02.946098 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:02.950448 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:02.950463 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:02.950482 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:02.950527 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:02.954066 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.954949 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.954957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.954960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.955102 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:02.955478 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:02.955481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:02.955483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:02.955872 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:02.956132 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:02.956437 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 23:23:02.956693 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":340,"threads_started":1}
I20260926 23:23:02.958596 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:03.044620 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 769
I20260926 23:23:03.060861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 408
I20260926 23:23:03.062111 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:03.172138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 408
2026-09-26T23:23:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4290 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:03 dist-test-slave-q5g8 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:03 dist-test-slave-q5g8 krb5kdc[919](info): set up 2 sockets
Sep 26 23:23:03 dist-test-slave-q5g8 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260926 23:23:04.289098 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:04 dist-test-slave-q5g8 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:04.485038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36213
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:04.823728 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:04.824123 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:04.824294 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:04.831542 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:04.831713 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:04.831827 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:04.831933 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:04.832026 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:04.842011 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:04.843458 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:04.844911 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:04.857327 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:04.857388 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:04.857288 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:04.858389 932 server_base.cc:1034] running on GCE node
I20260926 23:23:04.859378 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:04.861219 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:04.862458 932 hybrid_clock.cc:648] HybridClock initialized: now 1790464984862407 us; error 39 us; skew 500 ppm
I20260926 23:23:04.862881 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:04 dist-test-slave-q5g8 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:04.866564 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:04.868285 932 webserver.cc:460] Webserver started at http://127.0.97.62:36853/ using document root <none> and password file <none>
I20260926 23:23:04.869119 932 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:04.869226 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:04.869575 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:04.872751 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "2f8d81079aa0448ab87b4da3548c088e"
format_stamp: "Formatted at 2026-09-26 23:23:04 on dist-test-slave-q5g8"
server_key: "799b9b8acd5591fe22ff197e2fb5dcc0"
server_key_iv: "1b5840375bdeb9aa22211bb079568e63"
server_key_version: "encryptionkey@0"
I20260926 23:23:04.873522 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f8d81079aa0448ab87b4da3548c088e"
format_stamp: "Formatted at 2026-09-26 23:23:04 on dist-test-slave-q5g8"
server_key: "799b9b8acd5591fe22ff197e2fb5dcc0"
server_key_iv: "1b5840375bdeb9aa22211bb079568e63"
server_key_version: "encryptionkey@0"
I20260926 23:23:04.878811 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:04.882397 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:04.883806 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:23:04.883996 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "2f8d81079aa0448ab87b4da3548c088e"
format_stamp: "Formatted at 2026-09-26 23:23:04 on dist-test-slave-q5g8"
server_key: "799b9b8acd5591fe22ff197e2fb5dcc0"
server_key_iv: "1b5840375bdeb9aa22211bb079568e63"
server_key_version: "encryptionkey@0"
I20260926 23:23:04.884225 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:04.899847 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:04.904744 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:04.905103 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:04.925346 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36213
I20260926 23:23:04.925452 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36213 every 8 connection(s)
I20260926 23:23:04.926870 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:04.930963 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:04.933678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 932
I20260926 23:23:04.933943 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:04.934603 388 external_mini_cluster.cc:1428] Setting key 53b1b1a0e77fbbd408d53354059ff6ea
I20260926 23:23:04.945106 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e: Bootstrap starting.
I20260926 23:23:04.948894 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:04.949952 999 log.cc:824] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:23:04 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:04.953791 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e: No bootstrap required, opened a new log
I20260926 23:23:04.962056 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:04.936989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.0.97.62:36213)
0926 23:23:04.937920 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:04.937939 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:04.937995 (+ 56us) server_negotiation.cc:407] Connection header received
0926 23:23:04.940536 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:04.940570 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:04.941224 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:04.941762 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:04.943236 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:04.945295 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:04.946820 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:04.947885 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:04.948095 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:04.953159 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:04.953167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:04.953184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:04.953224 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:04.956907 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:04.957898 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:04.957906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:04.957914 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:04.958119 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:04.958580 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:04.958584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:04.958586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:04.959222 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:04.959522 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:04.960270 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 23:23:04.960811 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":420,"threads_started":1}
I20260926 23:23:04.963940 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8d81079aa0448ab87b4da3548c088e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36213 } }
I20260926 23:23:04.964380 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:04.964524 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d81079aa0448ab87b4da3548c088e, State: Initialized, Role: FOLLOWER
I20260926 23:23:04.965201 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8d81079aa0448ab87b4da3548c088e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36213 } }
I20260926 23:23:04.965497 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:04.965658 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:04.965852 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:04.968238 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8d81079aa0448ab87b4da3548c088e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36213 } }
I20260926 23:23:04.968909 999 leader_election.cc:302] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f8d81079aa0448ab87b4da3548c088e; no voters:
I20260926 23:23:04.969630 999 leader_election.cc:288] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:04.969817 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:04.971129 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [term 1 LEADER]: Becoming Leader. State: Replica: 2f8d81079aa0448ab87b4da3548c088e, State: Running, Role: LEADER
I20260926 23:23:04.971733 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8d81079aa0448ab87b4da3548c088e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36213 } }
I20260926 23:23:04.972594 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:04.975854 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f8d81079aa0448ab87b4da3548c088e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8d81079aa0448ab87b4da3548c088e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36213 } } }
I20260926 23:23:04.975888 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8d81079aa0448ab87b4da3548c088e. Latest consensus state: current_term: 1 leader_uuid: "2f8d81079aa0448ab87b4da3548c088e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8d81079aa0448ab87b4da3548c088e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36213 } } }
I20260926 23:23:04.976464 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:04.976464 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:04.979315 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:04.984988 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:04.995925 1013 catalog_manager.cc:1347] Generated new cluster ID: 0e6ba1ae6e5942ecbd3cf01714f3f4ba
I20260926 23:23:04.996091 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:05.012667 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:05.014137 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:05.022826 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f8d81079aa0448ab87b4da3548c088e: Generated new TSK 0
I20260926 23:23:05.023761 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:05.086062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36213
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:05.449412 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:05.449736 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:05.449895 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:05.457728 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:05.457863 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:05.458047 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:05.468369 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36213
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:05.469928 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:05.471442 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:05.486809 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:05.486946 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:05.488137 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:05.489223 1025 server_base.cc:1034] running on GCE node
I20260926 23:23:05.489964 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:05.491076 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:05.492327 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790464985492243 us; error 87 us; skew 500 ppm
I20260926 23:23:05.492759 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:05 dist-test-slave-q5g8 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:05.496261 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:05.498016 1025 webserver.cc:460] Webserver started at http://127.0.97.1:35841/ using document root <none> and password file <none>
I20260926 23:23:05.498804 1025 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:05.498986 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:05.499342 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:05.502737 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "958e3f2ca337475a9da0d9c3bb3d3383"
format_stamp: "Formatted at 2026-09-26 23:23:05 on dist-test-slave-q5g8"
server_key: "81cf5ed725c5e909d2e290d70f082278"
server_key_iv: "7aeda73e49e3615dd683a3d4b9675cc5"
server_key_version: "encryptionkey@0"
I20260926 23:23:05.503687 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "958e3f2ca337475a9da0d9c3bb3d3383"
format_stamp: "Formatted at 2026-09-26 23:23:05 on dist-test-slave-q5g8"
server_key: "81cf5ed725c5e909d2e290d70f082278"
server_key_iv: "7aeda73e49e3615dd683a3d4b9675cc5"
server_key_version: "encryptionkey@0"
I20260926 23:23:05.510288 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:23:05.514449 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:05.516080 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:23:05.516297 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "958e3f2ca337475a9da0d9c3bb3d3383"
format_stamp: "Formatted at 2026-09-26 23:23:05 on dist-test-slave-q5g8"
server_key: "81cf5ed725c5e909d2e290d70f082278"
server_key_iv: "7aeda73e49e3615dd683a3d4b9675cc5"
server_key_version: "encryptionkey@0"
I20260926 23:23:05.516575 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:05.543978 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:05.548851 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:05.549170 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:05.550233 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:05.551667 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:05.551774 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:05.551925 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:05.551998 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:05.588927 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38279
I20260926 23:23:05.588979 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38279 every 8 connection(s)
I20260926 23:23:05.590445 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:05.598734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 1025
I20260926 23:23:05.598974 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:05.599773 388 external_mini_cluster.cc:1428] Setting key abe574fd0fefc323f8c8bafd25220852
Sep 26 23:23:05 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:05.613952 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:05.593516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44435 (local address 127.0.97.62:36213)
0926 23:23:05.594019 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:05.594024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:05.595140 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 23:23:05.597224 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:05.597228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:05.597306 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:05.597415 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:05.599607 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:05.600501 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:05.602475 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:05.602873 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:05.603018 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:05.607287 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:05.607293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:05.607296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:05.607334 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:05.610155 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:05.611093 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:05.611099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:05.611101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:05.611235 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:05.611780 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:05.611792 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:05.611796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:05.612122 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:05.612300 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:05.613492 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0926 23:23:05.613682 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
I20260926 23:23:05.615864 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:05.594153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36213 (local address 127.0.97.1:44435)
0926 23:23:05.594945 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:05.594989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:05.596914 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:05.597657 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:05.597666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:05.598357 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:05.599369 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:05.599402 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:05.600768 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:05.600772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:05.602287 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:05.602306 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:05.603123 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:05.603132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:05.603393 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:05.604199 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:05.604224 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:05.606999 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:05.610372 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:05.610383 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:05.610405 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:05.610928 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:05.611385 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:05.611390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:05.611394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:05.611641 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:05.612326 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:05.612340 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:05.612943 (+ 603us) client_negotiation.cc:769] Sending connection context
0926 23:23:05.613347 (+ 404us) client_negotiation.cc:240] Negotiation successful
0926 23:23:05.614163 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:23:05.617755 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36213
I20260926 23:23:05.618111 1153 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:05.619043 1153 heartbeater.cc:511] Master 127.0.97.62:36213 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:23:05.621793 964 ts_manager.cc:194] Registered new tserver with Master: 958e3f2ca337475a9da0d9c3bb3d3383 (127.0.97.1:38279)
I20260926 23:23:05.623368 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:05.648231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36213
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:05.997766 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:05.998142 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:05.998327 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:06.006098 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:06.006299 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:06.006486 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:06.017716 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36213
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:06.019198 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:06.020685 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:06.035122 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:06.035247 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:06.035377 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:06.036342 1159 server_base.cc:1034] running on GCE node
I20260926 23:23:06.037210 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:06.038511 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:06.039779 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790464986039720 us; error 60 us; skew 500 ppm
I20260926 23:23:06.040182 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:06 dist-test-slave-q5g8 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:06.044312 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:06.046452 1159 webserver.cc:460] Webserver started at http://127.0.97.2:39169/ using document root <none> and password file <none>
I20260926 23:23:06.047371 1159 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:06.047570 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:06.047966 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:06.051688 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "c3a964e6f2394414b09833a3397e714b"
format_stamp: "Formatted at 2026-09-26 23:23:06 on dist-test-slave-q5g8"
server_key: "aa7992bf97abe42c22c12c79930dde50"
server_key_iv: "f9274adca99b6c95da20414747b01336"
server_key_version: "encryptionkey@0"
I20260926 23:23:06.052711 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c3a964e6f2394414b09833a3397e714b"
format_stamp: "Formatted at 2026-09-26 23:23:06 on dist-test-slave-q5g8"
server_key: "aa7992bf97abe42c22c12c79930dde50"
server_key_iv: "f9274adca99b6c95da20414747b01336"
server_key_version: "encryptionkey@0"
I20260926 23:23:06.059319 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:23:06.063720 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:06.065475 1159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:23:06.065716 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "c3a964e6f2394414b09833a3397e714b"
format_stamp: "Formatted at 2026-09-26 23:23:06 on dist-test-slave-q5g8"
server_key: "aa7992bf97abe42c22c12c79930dde50"
server_key_iv: "f9274adca99b6c95da20414747b01336"
server_key_version: "encryptionkey@0"
I20260926 23:23:06.065989 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:06.078310 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:06.083168 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:06.083479 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:06.084568 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:06.086349 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:06.086474 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:06.086580 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:06.086629 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:06.123656 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45959
I20260926 23:23:06.123730 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45959 every 8 connection(s)
I20260926 23:23:06.125234 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:06.127422 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 1159
I20260926 23:23:06.127647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:06.128131 388 external_mini_cluster.cc:1428] Setting key 8053b895bd81ce0608eb0653b927f47a
Sep 26 23:23:06 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:23:06.151739 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.128316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39427 (local address 127.0.97.62:36213)
0926 23:23:06.128836 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.128843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.130000 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 23:23:06.132910 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.132914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.132989 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.133094 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:06.135624 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.136559 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.138380 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.139222 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.139379 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.143671 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:06.143681 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:06.143690 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:06.143745 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:06.147714 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:06.148668 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:06.148673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:06.148676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:06.148898 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:06.149550 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:06.149554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:06.149556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:06.149861 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:06.150047 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.150956 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.151254 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":226,"threads_started":1}
I20260926 23:23:06.153075 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.128954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36213 (local address 127.0.97.2:39427)
0926 23:23:06.129821 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:06.129876 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:06.132499 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:06.133444 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:06.133462 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:06.134423 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:06.135389 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.135421 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.136804 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.136808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.138172 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.138213 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.139464 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.139469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.139694 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.140482 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:06.140507 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:06.143346 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:06.147961 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:06.147969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:06.147986 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:06.148503 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:06.149108 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:06.149116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:06.149123 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:06.149384 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:06.150012 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:06.150056 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:06.150486 (+ 430us) client_negotiation.cc:769] Sending connection context
0926 23:23:06.150754 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 23:23:06.151338 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":419,"threads_started":1}
I20260926 23:23:06.154899 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36213
I20260926 23:23:06.155284 1287 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:06.156129 1287 heartbeater.cc:511] Master 127.0.97.62:36213 requested a full tablet report, sending...
I20260926 23:23:06.158145 964 ts_manager.cc:194] Registered new tserver with Master: c3a964e6f2394414b09833a3397e714b (127.0.97.2:45959)
I20260926 23:23:06.159082 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:06.180899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36213
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:06.530828 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:06.531157 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:06.531330 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:06.538926 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:06.539060 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:06.539265 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:06.550179 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36213
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:06.551626 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:06.553123 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:06.568846 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:06.568904 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:06.569785 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:06.569917 1293 server_base.cc:1034] running on GCE node
I20260926 23:23:06.570866 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:06.572036 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:06.573338 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790464986573253 us; error 65 us; skew 500 ppm
I20260926 23:23:06.573777 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:06 dist-test-slave-q5g8 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:06.577519 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:06.579296 1293 webserver.cc:460] Webserver started at http://127.0.97.3:37681/ using document root <none> and password file <none>
I20260926 23:23:06.580128 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:06.580291 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:06.580634 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:06.583889 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "abab80784ed5455daf9ab2ad8e7fc93a"
format_stamp: "Formatted at 2026-09-26 23:23:06 on dist-test-slave-q5g8"
server_key: "1be09ab9f5d5e1898fee86f637517424"
server_key_iv: "4ee4d93ec7d466ee783f2f8fef776bea"
server_key_version: "encryptionkey@0"
I20260926 23:23:06.584875 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "abab80784ed5455daf9ab2ad8e7fc93a"
format_stamp: "Formatted at 2026-09-26 23:23:06 on dist-test-slave-q5g8"
server_key: "1be09ab9f5d5e1898fee86f637517424"
server_key_iv: "4ee4d93ec7d466ee783f2f8fef776bea"
server_key_version: "encryptionkey@0"
I20260926 23:23:06.591272 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:23:06.596073 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:06.597913 1293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 23:23:06.598151 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "abab80784ed5455daf9ab2ad8e7fc93a"
format_stamp: "Formatted at 2026-09-26 23:23:06 on dist-test-slave-q5g8"
server_key: "1be09ab9f5d5e1898fee86f637517424"
server_key_iv: "4ee4d93ec7d466ee783f2f8fef776bea"
server_key_version: "encryptionkey@0"
I20260926 23:23:06.598403 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:06.625653 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:06.626703 1153 heartbeater.cc:503] Master 127.0.97.62:36213 was elected leader, sending a full tablet report...
I20260926 23:23:06.631163 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:06.631424 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:06.632444 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:06.634030 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:06.634116 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:06.634207 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:06.634254 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:06.671439 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43619
I20260926 23:23:06.671504 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43619 every 8 connection(s)
I20260926 23:23:06.673028 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:06.680783 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 1293
I20260926 23:23:06.681001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:06.681565 388 external_mini_cluster.cc:1428] Setting key 31cab093dfffcba3a5c4acdc1d7b5e0e
Sep 26 23:23:06 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:06.698437 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.675904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53817 (local address 127.0.97.62:36213)
0926 23:23:06.676370 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.676374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.677586 (+ 1212us) server_negotiation.cc:407] Connection header received
0926 23:23:06.679693 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.679699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.679817 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.679962 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:06.682368 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.683611 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.686117 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.686564 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.686677 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.691431 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:06.691439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:06.691451 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:06.691511 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:06.694554 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:06.695495 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:06.695499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:06.695501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:06.695677 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:06.696210 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:06.696213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:06.696216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:06.696619 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:06.696831 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.697792 (+ 961us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.698018 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260926 23:23:06.699667 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.676537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36213 (local address 127.0.97.3:53817)
0926 23:23:06.677398 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:06.677449 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:06.679361 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:06.680236 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:06.680246 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:06.681162 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:06.682131 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.682165 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.683865 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.683872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.685956 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.685972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.686847 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.686855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.687131 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.688121 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:06.688151 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:06.690970 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:06.694766 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:06.694775 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:06.694793 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:06.695352 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:06.695797 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:06.695801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:06.695803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:06.696060 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:06.696799 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:06.696810 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:06.697307 (+ 497us) client_negotiation.cc:769] Sending connection context
0926 23:23:06.697665 (+ 358us) client_negotiation.cc:240] Negotiation successful
0926 23:23:06.698360 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":348,"threads_started":1}
I20260926 23:23:06.701341 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36213
I20260926 23:23:06.701699 1421 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:06.702555 1421 heartbeater.cc:511] Master 127.0.97.62:36213 requested a full tablet report, sending...
I20260926 23:23:06.704311 964 ts_manager.cc:194] Registered new tserver with Master: abab80784ed5455daf9ab2ad8e7fc93a (127.0.97.3:43619)
I20260926 23:23:06.705267 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53817
I20260926 23:23:06.710765 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:23:06.731968 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.717254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.97.62:36213)
0926 23:23:06.717535 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.717542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.717919 (+ 377us) server_negotiation.cc:407] Connection header received
0926 23:23:06.718062 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.718066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.718135 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.718243 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:06.720413 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.721279 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.722667 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.723108 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.723225 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.725946 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:06.725952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:06.725956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:06.725999 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:06.729094 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:06.729932 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:06.729937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:06.729939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:06.730083 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:06.730468 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:06.730472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:06.730474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:06.730841 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:06.731040 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.731349 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.731547 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 23:23:06.739459 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:06.747572 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:06.775280 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.762277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38279 (local address 127.0.0.1:54590)
0926 23:23:06.764015 (+ 1738us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:06.764067 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:06.764317 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:06.766116 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:06.766126 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:06.766157 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:06.766640 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.766655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.769565 (+ 2910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.769570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.772429 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.772443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.774144 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.774151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.774356 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.774422 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 23:23:06.774588 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 23:23:06.774818 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":1,"thread_start_us":506,"threads_started":1}
I20260926 23:23:06.775553 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.762977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.0.97.2:45959)
0926 23:23:06.764912 (+ 1935us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.764925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.765351 (+ 426us) server_negotiation.cc:407] Connection header received
0926 23:23:06.765623 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.765629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.765852 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.766032 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:06.767134 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.769365 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.772802 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.774383 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.774541 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.774753 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.775007 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.775195 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":1618,"threads_started":1}
I20260926 23:23:06.776046 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.762466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.0.97.1:38279)
0926 23:23:06.765425 (+ 2959us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.765435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.765457 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:23:06.765568 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.765577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.765852 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.766938 (+ 1086us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:06.767217 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.769365 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.772627 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.773901 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.774074 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.774351 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.775551 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.775735 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2795,"thread_start_us":2630,"threads_started":1}
I20260926 23:23:06.777719 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.762483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45959 (local address 127.0.0.1:50046)
0926 23:23:06.765230 (+ 2747us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:06.765255 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:06.765456 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:06.766357 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:06.766360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:06.766380 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:06.766926 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.766938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.769565 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.769570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.772573 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.772590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.774590 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.774597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.774785 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.774818 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:23:06.776034 (+ 1216us) client_negotiation.cc:240] Negotiation successful
0926 23:23:06.776845 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2622,"mutex_wait_us":688,"thread_start_us":2461,"threads_started":1}
I20260926 23:23:06.780454 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb8b2ffe8b24ebebe523ffd7ce30946 (DEFAULT_TABLE table=test-table [id=76822bffbdc74e19b3c65610b60c9732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:06.781371 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.765419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43619 (local address 127.0.0.1:54294)
0926 23:23:06.771158 (+ 5739us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:06.771185 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:06.771340 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:06.772274 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:06.772279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:06.772309 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:06.772835 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.772847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.775501 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.775506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.778917 (+ 3411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.778932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.780512 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.780518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.780735 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.780775 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 23:23:06.780889 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:23:06.781023 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5587,"thread_start_us":5406,"threads_started":1}
I20260926 23:23:06.781764 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.765847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.0.97.3:43619)
0926 23:23:06.770206 (+ 4359us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.770215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.771486 (+ 1271us) server_negotiation.cc:407] Connection header received
0926 23:23:06.771597 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.771606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.771897 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.772109 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:06.773067 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.775366 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.779278 (+ 3912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.780299 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.780460 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.780730 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.780943 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.781197 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4106,"thread_start_us":3933,"threads_started":1}
I20260926 23:23:06.782068 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb8b2ffe8b24ebebe523ffd7ce30946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:06.783681 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb8b2ffe8b24ebebe523ffd7ce30946 (DEFAULT_TABLE table=test-table [id=76822bffbdc74e19b3c65610b60c9732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:06.785295 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb8b2ffe8b24ebebe523ffd7ce30946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:06.785039 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb8b2ffe8b24ebebe523ffd7ce30946 (DEFAULT_TABLE table=test-table [id=76822bffbdc74e19b3c65610b60c9732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:06.786607 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb8b2ffe8b24ebebe523ffd7ce30946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:06.799700 1440 tablet_bootstrap.cc:492] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383: Bootstrap starting.
I20260926 23:23:06.805794 1441 tablet_bootstrap.cc:492] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a: Bootstrap starting.
I20260926 23:23:06.807440 1442 tablet_bootstrap.cc:492] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b: Bootstrap starting.
I20260926 23:23:06.807523 1440 tablet_bootstrap.cc:654] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:06.809060 1440 log.cc:824] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:06.809878 1441 tablet_bootstrap.cc:654] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:06.811296 1441 log.cc:824] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:06.811901 1442 tablet_bootstrap.cc:654] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:06.813257 1440 tablet_bootstrap.cc:492] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383: No bootstrap required, opened a new log
I20260926 23:23:06.813608 1440 ts_tablet_manager.cc:1397] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260926 23:23:06.813908 1442 log.cc:824] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:06.815478 1441 tablet_bootstrap.cc:492] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a: No bootstrap required, opened a new log
I20260926 23:23:06.815800 1441 ts_tablet_manager.cc:1397] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 23:23:06.820597 1442 tablet_bootstrap.cc:492] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b: No bootstrap required, opened a new log
I20260926 23:23:06.821017 1442 ts_tablet_manager.cc:1397] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260926 23:23:06.824448 1441 raft_consensus.cc:348] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.824959 1441 raft_consensus.cc:374] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:06.825062 1441 raft_consensus.cc:729] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab80784ed5455daf9ab2ad8e7fc93a, State: Initialized, Role: FOLLOWER
I20260926 23:23:06.825802 1441 consensus_queue.cc:260] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.826301 1440 raft_consensus.cc:348] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.826776 1440 raft_consensus.cc:374] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:06.826900 1440 raft_consensus.cc:729] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958e3f2ca337475a9da0d9c3bb3d3383, State: Initialized, Role: FOLLOWER
I20260926 23:23:06.827365 1421 heartbeater.cc:503] Master 127.0.97.62:36213 was elected leader, sending a full tablet report...
I20260926 23:23:06.827859 1441 ts_tablet_manager.cc:1428] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 23:23:06.828656 1440 consensus_queue.cc:260] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.833287 1440 ts_tablet_manager.cc:1428] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260926 23:23:06.835532 1442 raft_consensus.cc:348] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.836014 1442 raft_consensus.cc:374] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:06.836146 1442 raft_consensus.cc:729] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a964e6f2394414b09833a3397e714b, State: Initialized, Role: FOLLOWER
I20260926 23:23:06.836944 1442 consensus_queue.cc:260] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.838332 1287 heartbeater.cc:503] Master 127.0.97.62:36213 was elected leader, sending a full tablet report...
I20260926 23:23:06.839110 1442 ts_tablet_manager.cc:1428] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
W20260926 23:23:06.843124 1154 tablet.cc:2369] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:23:06.877875 1288 tablet.cc:2369] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:23:06.924973 1422 tablet.cc:2369] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:06.942541 1448 raft_consensus.cc:482] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:06.942924 1448 raft_consensus.cc:504] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.945124 1448 leader_election.cc:288] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 958e3f2ca337475a9da0d9c3bb3d3383 (127.0.97.1:38279), abab80784ed5455daf9ab2ad8e7fc93a (127.0.97.3:43619)
I20260926 23:23:06.952147 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.945597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57343 (local address 127.0.97.1:38279)
0926 23:23:06.945854 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.945865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.946693 (+ 828us) server_negotiation.cc:407] Connection header received
0926 23:23:06.946890 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.946894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.946953 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.947055 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:06.947728 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.948447 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.950302 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.951112 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.951284 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.951394 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.951716 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.951862 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:23:06.952168 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.945680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45819 (local address 127.0.97.3:43619)
0926 23:23:06.945888 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:06.945897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:06.946598 (+ 701us) server_negotiation.cc:407] Connection header received
0926 23:23:06.946824 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:06.946828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:06.946899 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:06.947004 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:06.947589 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.948321 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.950309 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.951095 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.951255 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.951384 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:06.951706 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:23:06.951847 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:23:06.952485 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.945550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38279 (local address 127.0.97.2:57343)
0926 23:23:06.946576 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:06.946591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:06.946710 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:06.947068 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:06.947072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:06.947094 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:06.947568 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.947576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.948599 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.948605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.950123 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.950137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.951377 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.951384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.951554 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.951577 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:23:06.951688 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:23:06.951851 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":54,"thread_start_us":479,"threads_started":1}
I20260926 23:23:06.952657 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:06.945546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43619 (local address 127.0.97.2:45819)
0926 23:23:06.946466 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:06.946481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:06.946680 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:06.947019 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:06.947023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:06.947046 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:06.947433 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.947445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.948462 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.948469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.950126 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:06.950138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:06.951344 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:06.951353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:06.951523 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:06.951552 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 23:23:06.951666 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:23:06.951850 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":54,"thread_start_us":555,"threads_started":1}
I20260926 23:23:06.953838 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb8b2ffe8b24ebebe523ffd7ce30946" candidate_uuid: "c3a964e6f2394414b09833a3397e714b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" is_pre_election: true
I20260926 23:23:06.953860 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb8b2ffe8b24ebebe523ffd7ce30946" candidate_uuid: "c3a964e6f2394414b09833a3397e714b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" is_pre_election: true
I20260926 23:23:06.954540 1108 raft_consensus.cc:2454] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a964e6f2394414b09833a3397e714b in term 0.
I20260926 23:23:06.954566 1376 raft_consensus.cc:2454] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a964e6f2394414b09833a3397e714b in term 0.
I20260926 23:23:06.955672 1176 leader_election.cc:302] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 958e3f2ca337475a9da0d9c3bb3d3383, c3a964e6f2394414b09833a3397e714b; no voters:
I20260926 23:23:06.956216 1448 raft_consensus.cc:2781] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:06.956386 1448 raft_consensus.cc:482] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:06.956465 1448 raft_consensus.cc:3037] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:06.959128 1448 raft_consensus.cc:504] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.959791 1448 leader_election.cc:288] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [CANDIDATE]: Term 1 election: Requested vote from peers 958e3f2ca337475a9da0d9c3bb3d3383 (127.0.97.1:38279), abab80784ed5455daf9ab2ad8e7fc93a (127.0.97.3:43619)
I20260926 23:23:06.960770 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb8b2ffe8b24ebebe523ffd7ce30946" candidate_uuid: "c3a964e6f2394414b09833a3397e714b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abab80784ed5455daf9ab2ad8e7fc93a"
I20260926 23:23:06.960883 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb8b2ffe8b24ebebe523ffd7ce30946" candidate_uuid: "c3a964e6f2394414b09833a3397e714b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958e3f2ca337475a9da0d9c3bb3d3383"
I20260926 23:23:06.961043 1376 raft_consensus.cc:3037] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:06.961156 1108 raft_consensus.cc:3037] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:06.965020 1376 raft_consensus.cc:2454] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P abab80784ed5455daf9ab2ad8e7fc93a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a964e6f2394414b09833a3397e714b in term 1.
I20260926 23:23:06.965631 1175 leader_election.cc:302] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abab80784ed5455daf9ab2ad8e7fc93a, c3a964e6f2394414b09833a3397e714b; no voters:
I20260926 23:23:06.965662 1108 raft_consensus.cc:2454] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P 958e3f2ca337475a9da0d9c3bb3d3383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a964e6f2394414b09833a3397e714b in term 1.
I20260926 23:23:06.965972 1448 raft_consensus.cc:2781] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:06.966629 1448 raft_consensus.cc:686] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [term 1 LEADER]: Becoming Leader. State: Replica: c3a964e6f2394414b09833a3397e714b, State: Running, Role: LEADER
I20260926 23:23:06.967254 1448 consensus_queue.cc:237] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } }
I20260926 23:23:06.972728 963 catalog_manager.cc:5495] T 0bb8b2ffe8b24ebebe523ffd7ce30946 P c3a964e6f2394414b09833a3397e714b reported cstate change: term changed from 0 to 1, leader changed from <none> to c3a964e6f2394414b09833a3397e714b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c3a964e6f2394414b09833a3397e714b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958e3f2ca337475a9da0d9c3bb3d3383" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3a964e6f2394414b09833a3397e714b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abab80784ed5455daf9ab2ad8e7fc93a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43619 } health_report { overall_health: UNKNOWN } } }
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:07.034340 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:07.016643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.0.97.1:38279)
0926 23:23:07.016917 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:07.016926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:07.017483 (+ 557us) server_negotiation.cc:407] Connection header received
0926 23:23:07.017628 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:07.017632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:07.017724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:07.017816 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:07.019404 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.020188 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.021958 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.022982 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.023111 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:07.027891 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:07.027910 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:07.027929 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:07.027975 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:07.031154 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.032019 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.032028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.032030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.032172 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.032612 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.032617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.032619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.033070 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:07.033407 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:07.033731 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 23:23:07.033955 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:07.085392 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:07.068983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.0.97.1:38279)
0926 23:23:07.069207 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:07.069216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:07.069482 (+ 266us) server_negotiation.cc:407] Connection header received
0926 23:23:07.069609 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:07.069613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:07.069680 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:07.069856 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:07.071426 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.072149 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.073914 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.074390 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.074555 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:07.079051 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:07.079057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:07.079061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:07.079101 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:07.082123 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.082948 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.082954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.082957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.083114 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.083684 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.083697 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.083700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.084036 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:07.084342 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:07.084721 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 23:23:07.084978 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:07.137746 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:07.119775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.0.97.1:38279)
0926 23:23:07.120074 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:07.120081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:07.120403 (+ 322us) server_negotiation.cc:407] Connection header received
0926 23:23:07.120550 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:07.120554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:07.120623 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:07.120810 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:07.122282 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.123018 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.124458 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.124989 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.125140 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:07.130353 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:07.130361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:07.130365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:07.130416 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:07.133858 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.134946 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.134952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.134955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.135106 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.135585 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.135588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.135591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.136472 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:07.136790 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:07.137151 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 23:23:07.137416 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 23:23:07.139591 1068 tablet_service.cc:2908] Rejecting scan request for tablet 0bb8b2ffe8b24ebebe523ffd7ce30946: Uninitialized: clean time has not yet been initialized
I20260926 23:23:07.144018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1025
I20260926 23:23:07.158458 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:07.145475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.97.1:38279)
0926 23:23:07.145702 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:07.145709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:07.145848 (+ 139us) server_negotiation.cc:407] Connection header received
0926 23:23:07.145996 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:07.145999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:07.146079 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:07.146167 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:07.147431 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.148138 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.149567 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.150041 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.150162 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:07.152821 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:07.152827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:07.152831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:07.152883 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:07.155597 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.156453 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.156457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.156460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.156607 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.157053 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.157056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.157058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.157357 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:07.157598 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:07.157927 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 23:23:07.158124 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:23:07.159337 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:07.255105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 1025
I20260926 23:23:07.272838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1159
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:07.288230 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:07.273714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.97.2:45959)
0926 23:23:07.273910 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:07.273916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:07.273936 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:23:07.274144 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:07.274150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:07.274241 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:07.274346 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:07.275494 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.276221 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.277608 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.278015 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.278164 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:07.282217 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:07.282233 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:07.282249 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:07.282286 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:07.285383 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.286121 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.286130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.286132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.286297 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.286680 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.286685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.286690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.287045 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:07.287359 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:07.287617 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 23:23:07.287881 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:23:07.289058 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:07.383690 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 1159
I20260926 23:23:07.383751 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:07.383215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38279 (local address 127.0.97.2:41463)
0926 23:23:07.383413 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:07.383570 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:23:07.402299 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1293
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:07.418234 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:07.403121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.0.97.3:43619)
0926 23:23:07.403367 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:07.403371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:07.403393 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:23:07.403590 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:07.403594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:07.403671 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:07.403767 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:07.405263 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.405974 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.407294 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:07.407692 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:07.407790 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:07.412234 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:07.412250 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:07.412320 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:07.412361 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:07.415195 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.416028 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.416036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.416039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.416183 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:07.416552 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:07.416555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:07.416558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:07.417223 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:07.417482 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:07.417770 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 23:23:07.417948 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:23:07.418991 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:07.508749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 1293
I20260926 23:23:07.525475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 932
I20260926 23:23:07.526659 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:07.638432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 932
2026-09-26T23:23:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4476 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[1470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[1470](info): set up 2 sockets
Sep 26 23:23:07 dist-test-slave-q5g8 krb5kdc[1470](info): commencing operation
krb5kdc: starting...
W20260926 23:23:08.756918 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:08 dist-test-slave-q5g8 krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:08.945133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33425
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:09.279446 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:09.279760 1483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:09.279850 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:09.287230 1483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:09.287364 1483 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:23:09.287406 1483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:09.287462 1483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:09.287513 1483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:09.287576 1483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:09.298079 1483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33425
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:09.299754 1483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:09.301244 1483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:09.314316 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:09.314343 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:09.314697 1491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:09.315028 1483 server_base.cc:1034] running on GCE node
I20260926 23:23:09.315920 1483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:09.317655 1483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:09.318903 1483 hybrid_clock.cc:648] HybridClock initialized: now 1790464989318835 us; error 58 us; skew 500 ppm
I20260926 23:23:09.319253 1483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:09 dist-test-slave-q5g8 krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:09.322625 1483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:09.324146 1483 webserver.cc:460] Webserver started at http://127.0.97.62:38571/ using document root <none> and password file <none>
I20260926 23:23:09.325006 1483 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:09.325145 1483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:09.325481 1483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:09.328621 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "6f2711fdac6e41edaec5d683bbb8fc8a"
format_stamp: "Formatted at 2026-09-26 23:23:09 on dist-test-slave-q5g8"
server_key: "bedb9ad0eeac9f011b98149c95541aa0"
server_key_iv: "d681ec2f2a90a1b5b14846f10f872dd1"
server_key_version: "encryptionkey@0"
I20260926 23:23:09.329589 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "6f2711fdac6e41edaec5d683bbb8fc8a"
format_stamp: "Formatted at 2026-09-26 23:23:09 on dist-test-slave-q5g8"
server_key: "bedb9ad0eeac9f011b98149c95541aa0"
server_key_iv: "d681ec2f2a90a1b5b14846f10f872dd1"
server_key_version: "encryptionkey@0"
I20260926 23:23:09.335139 1483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:09.338989 1498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:09.340442 1483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:23:09.340672 1483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "6f2711fdac6e41edaec5d683bbb8fc8a"
format_stamp: "Formatted at 2026-09-26 23:23:09 on dist-test-slave-q5g8"
server_key: "bedb9ad0eeac9f011b98149c95541aa0"
server_key_iv: "d681ec2f2a90a1b5b14846f10f872dd1"
server_key_version: "encryptionkey@0"
I20260926 23:23:09.340993 1483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:09.362205 1483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:09.366586 1483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:09.366882 1483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:09.388823 1483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33425
I20260926 23:23:09.388788 1559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33425 every 8 connection(s)
I20260926 23:23:09.390201 1483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:09.393415 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 1483
I20260926 23:23:09.393644 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:09.394119 1560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:09.394299 388 external_mini_cluster.cc:1428] Setting key 94f1b0fac486b52b31b23eb6bf7e308a
I20260926 23:23:09.409106 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a: Bootstrap starting.
Sep 26 23:23:09 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:09.413004 1560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:09.414124 1560 log.cc:824] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:09.417454 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a: No bootstrap required, opened a new log
I20260926 23:23:09.418314 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:09.396221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.0.97.62:33425)
0926 23:23:09.397063 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:09.397078 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:09.397133 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:23:09.399271 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:09.399302 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:09.399882 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:09.400378 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:09.401780 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:09.403784 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:09.405130 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:09.406265 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:09.406454 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:09.410839 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:09.410850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:09.410865 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:09.410908 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:09.414103 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:09.415003 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:09.415014 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:09.415028 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:09.415272 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:09.415679 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:09.415683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:09.415685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:09.416173 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:09.416378 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:09.416885 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 23:23:09.417275 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":389,"threads_started":1}
I20260926 23:23:09.424870 1560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33425 } }
I20260926 23:23:09.425238 1560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:09.425361 1560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2711fdac6e41edaec5d683bbb8fc8a, State: Initialized, Role: FOLLOWER
I20260926 23:23:09.426017 1560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33425 } }
I20260926 23:23:09.426250 1560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:09.426375 1560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:09.426563 1560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:09.428838 1560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33425 } }
I20260926 23:23:09.429370 1560 leader_election.cc:302] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f2711fdac6e41edaec5d683bbb8fc8a; no voters:
I20260926 23:23:09.429967 1560 leader_election.cc:288] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:09.430079 1565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:09.430811 1565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [term 1 LEADER]: Becoming Leader. State: Replica: 6f2711fdac6e41edaec5d683bbb8fc8a, State: Running, Role: LEADER
I20260926 23:23:09.431366 1565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33425 } }
I20260926 23:23:09.432129 1560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:09.434370 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33425 } } }
I20260926 23:23:09.434341 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2711fdac6e41edaec5d683bbb8fc8a. Latest consensus state: current_term: 1 leader_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2711fdac6e41edaec5d683bbb8fc8a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33425 } } }
I20260926 23:23:09.434785 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:09.434785 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:09.437248 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:09.443391 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:09.453248 1572 catalog_manager.cc:1347] Generated new cluster ID: e1e767a2498c49e29753eb06c7273429
I20260926 23:23:09.453383 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:09.487635 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:09.489104 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:09.502568 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a: Generated new TSK 0
I20260926 23:23:09.503557 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:09.553783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33425
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:09.887569 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:09.887876 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:09.888042 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:09.895318 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:09.895431 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:09.895628 1586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:23:09.895751 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:09.905747 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:09.907045 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:09.908408 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:09.922060 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:09.922128 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:09.922346 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:09.923189 1586 server_base.cc:1034] running on GCE node
I20260926 23:23:09.923852 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:09.924882 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:09.926102 1586 hybrid_clock.cc:648] HybridClock initialized: now 1790464989926040 us; error 36 us; skew 500 ppm
I20260926 23:23:09.926405 1586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:09 dist-test-slave-q5g8 krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:09.929600 1586 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:09.930996 1586 webserver.cc:460] Webserver started at http://127.0.97.1:34135/ using document root <none> and password file <none>
I20260926 23:23:09.931697 1586 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:09.931808 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:09.932142 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:09.935189 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "4b5ed0ec76ac46cf8879afa30108d38f"
format_stamp: "Formatted at 2026-09-26 23:23:09 on dist-test-slave-q5g8"
server_key: "6a25bc877e353afc980715464325d5dd"
server_key_iv: "1ba7c2c2338b062180103a2a59c66f68"
server_key_version: "encryptionkey@0"
I20260926 23:23:09.935981 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4b5ed0ec76ac46cf8879afa30108d38f"
format_stamp: "Formatted at 2026-09-26 23:23:09 on dist-test-slave-q5g8"
server_key: "6a25bc877e353afc980715464325d5dd"
server_key_iv: "1ba7c2c2338b062180103a2a59c66f68"
server_key_version: "encryptionkey@0"
I20260926 23:23:09.941701 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 23:23:09.945355 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:09.946781 1586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:09.946954 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "4b5ed0ec76ac46cf8879afa30108d38f"
format_stamp: "Formatted at 2026-09-26 23:23:09 on dist-test-slave-q5g8"
server_key: "6a25bc877e353afc980715464325d5dd"
server_key_iv: "1ba7c2c2338b062180103a2a59c66f68"
server_key_version: "encryptionkey@0"
I20260926 23:23:09.947180 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:09.961720 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:09.965772 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:09.966044 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:09.968730 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:09.968845 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 23:23:09.969017 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:09.969099 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:23:09 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:09.994431 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:09.969486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50649 (local address 127.0.97.62:33425)
0926 23:23:09.970010 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:09.970016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:09.971263 (+ 1247us) server_negotiation.cc:407] Connection header received
0926 23:23:09.973656 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:09.973662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:09.973748 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:09.973872 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:09.976122 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:09.977472 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:09.980185 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:09.981005 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:09.981177 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:09.986441 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:09.986446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:09.986449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:09.986510 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:09.989915 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:09.991219 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:09.991227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:09.991234 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:09.991404 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:09.992039 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:09.992047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:09.992051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:09.992470 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:09.992625 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:09.993823 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0926 23:23:09.994055 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":294,"threads_started":1}
I20260926 23:23:09.995471 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:09.970168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33425 (local address 127.0.97.1:50649)
0926 23:23:09.971074 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:09.971124 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:09.973332 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:09.974160 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:09.974175 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:09.975004 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:09.975890 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:09.975923 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:09.977656 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:09.977663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:09.979992 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:09.980010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:09.981287 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:09.981297 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:09.981609 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:09.982534 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:09.982567 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:09.986132 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:09.990094 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:09.990111 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:09.990141 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:09.990999 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:09.991551 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:09.991559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:09.991565 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:09.991888 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:09.992766 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:09.992777 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:09.993353 (+ 576us) client_negotiation.cc:769] Sending connection context
0926 23:23:09.993753 (+ 400us) client_negotiation.cc:240] Negotiation successful
0926 23:23:09.994295 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":402,"threads_started":1}
I20260926 23:23:10.014483 1586 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34455
I20260926 23:23:10.014498 1721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34455 every 8 connection(s)
I20260926 23:23:10.015832 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:10.017222 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.002161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45657 (local address 127.0.97.62:33425)
0926 23:23:10.002342 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:10.002346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:10.002971 (+ 625us) server_negotiation.cc:407] Connection header received
0926 23:23:10.003190 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:10.003196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:10.003266 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:10.003369 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:10.004605 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.005680 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.007839 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.008246 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.008375 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:10.011131 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:10.011138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:10.011146 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:10.011188 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:10.014304 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:10.015141 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:10.015149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:10.015154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:10.015295 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:10.015656 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:10.015662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:10.015666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:10.015987 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:10.016125 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:10.016601 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 23:23:10.016864 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:23:10.017355 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.002251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33425 (local address 127.0.97.1:45657)
0926 23:23:10.002861 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:10.002880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:10.003061 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:10.003398 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:10.003402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:10.003996 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:10.004454 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:10.004465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.005794 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.005800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:10.007683 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:10.007695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.008439 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.008446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:10.008622 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:10.009423 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:10.009460 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:10.010919 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:10.014445 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:10.014452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:10.014461 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:10.015013 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:10.015405 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:10.015412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:10.015415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:10.015546 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:10.016090 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:10.016095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:10.016298 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 23:23:10.016493 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 23:23:10.016864 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":250,"threads_started":1}
I20260926 23:23:10.018282 1722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33425
I20260926 23:23:10.018529 1722 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:10.019433 1722 heartbeater.cc:511] Master 127.0.97.62:33425 requested a full tablet report, sending...
I20260926 23:23:10.021804 1514 ts_manager.cc:194] Registered new tserver with Master: 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1:34455)
I20260926 23:23:10.023181 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50649
I20260926 23:23:10.023347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 1586
I20260926 23:23:10.023584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:10.024399 388 external_mini_cluster.cc:1428] Setting key 400f96ad541f10d6b22d3f6c690ffff7
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:10.066445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33425
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:10.399827 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:10.400125 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:10.400310 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:10.407975 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:10.408100 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:10.408249 1726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:23:10.408362 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:10.418397 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:10.419736 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:10.421164 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:10.434882 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:10.435011 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:10.434906 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:10.436280 1726 server_base.cc:1034] running on GCE node
I20260926 23:23:10.437004 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:10.438109 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:10.439332 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790464990439245 us; error 62 us; skew 500 ppm
I20260926 23:23:10.439670 1726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:10 dist-test-slave-q5g8 krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:10.442924 1726 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:10.444666 1726 webserver.cc:460] Webserver started at http://127.0.97.2:44467/ using document root <none> and password file <none>
I20260926 23:23:10.445580 1726 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:10.445703 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:10.446038 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:10.449090 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "8e3d423503db49ca9ff71314bd9be3dd"
format_stamp: "Formatted at 2026-09-26 23:23:10 on dist-test-slave-q5g8"
server_key: "789dfa5b25c08b26a5621783a676a47a"
server_key_iv: "6108d7678d0e14b2bca0b7d140193b95"
server_key_version: "encryptionkey@0"
I20260926 23:23:10.449884 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8e3d423503db49ca9ff71314bd9be3dd"
format_stamp: "Formatted at 2026-09-26 23:23:10 on dist-test-slave-q5g8"
server_key: "789dfa5b25c08b26a5621783a676a47a"
server_key_iv: "6108d7678d0e14b2bca0b7d140193b95"
server_key_version: "encryptionkey@0"
I20260926 23:23:10.455240 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:10.458930 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:10.460337 1726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:23:10.460525 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "8e3d423503db49ca9ff71314bd9be3dd"
format_stamp: "Formatted at 2026-09-26 23:23:10 on dist-test-slave-q5g8"
server_key: "789dfa5b25c08b26a5621783a676a47a"
server_key_iv: "6108d7678d0e14b2bca0b7d140193b95"
server_key_version: "encryptionkey@0"
I20260926 23:23:10.460811 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:10.476824 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:10.480938 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:10.481235 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:10.484136 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:10.484249 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:10.484453 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:10.484527 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:23:10 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:10.511966 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.485326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33425 (local address 127.0.97.2:59943)
0926 23:23:10.486132 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:10.486170 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:10.488093 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:10.489166 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:10.489178 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:10.490088 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:10.491224 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:10.491263 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.493012 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.493023 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:10.496088 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:10.496114 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.497127 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.497135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:10.497378 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:10.498297 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:10.498321 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:10.501598 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:10.505743 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:10.505752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:10.505768 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:10.506476 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:10.506989 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:10.506999 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:10.507004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:10.507334 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:10.508269 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:10.508285 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:10.508966 (+ 681us) client_negotiation.cc:769] Sending connection context
0926 23:23:10.509377 (+ 411us) client_negotiation.cc:240] Negotiation successful
0926 23:23:10.510161 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":383,"threads_started":1}
I20260926 23:23:10.512013 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.484614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59943 (local address 127.0.97.62:33425)
0926 23:23:10.484855 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:10.484864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:10.486300 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 23:23:10.488537 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:10.488546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:10.488662 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:10.488839 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:10.491501 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.492775 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.496304 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.496831 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.496978 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:10.501914 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:10.501920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:10.501924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:10.501961 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:10.505509 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:10.506634 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:10.506641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:10.506645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:10.506808 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:10.507546 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:10.507552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:10.507556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:10.508001 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:10.508166 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:10.511195 (+ 3029us) server_negotiation.cc:299] Negotiation successful
0926 23:23:10.511519 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:23:10.533367 1726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36225
I20260926 23:23:10.533459 1860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36225 every 8 connection(s)
I20260926 23:23:10.534734 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:10.535262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 1726
I20260926 23:23:10.535513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:10.536166 388 external_mini_cluster.cc:1428] Setting key 52b7d0710feaa10c8f483da98c5c8e50
I20260926 23:23:10.537674 1861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33425
I20260926 23:23:10.537995 1861 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:10.538434 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.520023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33425 (local address 127.0.97.2:43777)
0926 23:23:10.520951 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:10.520974 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:10.521195 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:10.521807 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:10.521815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:10.523143 (+ 1328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:10.523754 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:10.523778 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.525425 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.525434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:10.527886 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:10.527916 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.529186 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.529197 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:10.529450 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:10.530287 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:10.530317 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:10.531599 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:10.535319 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:10.535325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:10.535330 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:10.535814 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:10.536256 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:10.536261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:10.536265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:10.536415 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:10.537198 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:10.537202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:10.537420 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 23:23:10.537570 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 23:23:10.537939 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":586,"threads_started":1}
I20260926 23:23:10.539287 1861 heartbeater.cc:511] Master 127.0.97.62:33425 requested a full tablet report, sending...
I20260926 23:23:10.539525 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.520163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43777 (local address 127.0.97.62:33425)
0926 23:23:10.520433 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:10.520442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:10.521080 (+ 638us) server_negotiation.cc:407] Connection header received
0926 23:23:10.521362 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:10.521369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:10.521471 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:10.521592 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:10.524002 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.525200 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.528103 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:10.528912 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:10.529086 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:10.531859 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:10.531868 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:10.531877 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:10.531925 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:10.535165 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:10.535938 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:10.535944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:10.535950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:10.536109 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:10.536579 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:10.536586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:10.536591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:10.537070 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:10.538826 (+ 1756us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:10.538955 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 23:23:10.539183 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:23:10.541030 1514 ts_manager.cc:194] Registered new tserver with Master: 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2:36225)
I20260926 23:23:10.542275 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59943
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:10.580842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33425
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:10.910799 1865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:10.911192 1865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:10.911376 1865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:10.919080 1865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:10.919215 1865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:10.919308 1865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:23:10.919399 1865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:10.929718 1865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:10.930989 1865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:10.932353 1865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:10.946403 1873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:10.946650 1870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:10.946429 1871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:10.947204 1865 server_base.cc:1034] running on GCE node
I20260926 23:23:10.947857 1865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:10.948963 1865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:10.950232 1865 hybrid_clock.cc:648] HybridClock initialized: now 1790464990950162 us; error 41 us; skew 500 ppm
I20260926 23:23:10.950575 1865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:10 dist-test-slave-q5g8 krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:10.953914 1865 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:10.955556 1865 webserver.cc:460] Webserver started at http://127.0.97.3:37889/ using document root <none> and password file <none>
I20260926 23:23:10.956225 1865 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:10.956310 1865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:10.956573 1865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:10.959614 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "2cb61606105a45d3ba3a4cdfae8ec765"
format_stamp: "Formatted at 2026-09-26 23:23:10 on dist-test-slave-q5g8"
server_key: "f4cbb0a5b88c65345e30b2f3199e7fcd"
server_key_iv: "59f8d1bff0b3055edde6ace8fe2b4056"
server_key_version: "encryptionkey@0"
I20260926 23:23:10.960335 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2cb61606105a45d3ba3a4cdfae8ec765"
format_stamp: "Formatted at 2026-09-26 23:23:10 on dist-test-slave-q5g8"
server_key: "f4cbb0a5b88c65345e30b2f3199e7fcd"
server_key_iv: "59f8d1bff0b3055edde6ace8fe2b4056"
server_key_version: "encryptionkey@0"
I20260926 23:23:10.965641 1865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:23:10.969166 1880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:10.970541 1865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 23:23:10.970711 1865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "2cb61606105a45d3ba3a4cdfae8ec765"
format_stamp: "Formatted at 2026-09-26 23:23:10 on dist-test-slave-q5g8"
server_key: "f4cbb0a5b88c65345e30b2f3199e7fcd"
server_key_iv: "59f8d1bff0b3055edde6ace8fe2b4056"
server_key_version: "encryptionkey@0"
I20260926 23:23:10.970938 1865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:10.989349 1865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:10.993578 1865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:10.993856 1865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:10.996603 1865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:10.996760 1865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:10.996982 1865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:10.997089 1865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:23:11 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:11.025630 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.997092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33435 (local address 127.0.97.62:33425)
0926 23:23:10.997279 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:10.997285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:10.998766 (+ 1481us) server_negotiation.cc:407] Connection header received
0926 23:23:11.001039 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.001044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.001150 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.001263 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:11.003645 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.004790 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.008443 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.009054 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.009212 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.015949 (+ 6737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:11.015960 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:11.015968 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:11.016031 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:11.021895 (+ 5864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.023024 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.023030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.023035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.023176 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.023648 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.023654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.023657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.023996 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:11.024124 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.025032 (+ 908us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.025267 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:23:11.026381 1722 heartbeater.cc:503] Master 127.0.97.62:33425 was elected leader, sending a full tablet report...
I20260926 23:23:11.026748 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:10.997672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33425 (local address 127.0.97.3:33435)
0926 23:23:10.998568 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:10.998627 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.000664 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.001553 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.001569 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.002446 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:11.003350 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.003389 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.005067 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.005073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.008211 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.008232 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.009410 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.009423 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.009830 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.010800 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:11.010826 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:11.015565 (+ 4739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:11.022110 (+ 6545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:11.022124 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:11.022148 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:11.022863 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:11.023300 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:11.023306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:11.023309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:11.023517 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:11.024190 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:11.024196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:11.024555 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.024887 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.025475 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":379,"threads_started":1}
I20260926 23:23:11.048120 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.033616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33425 (local address 127.0.97.3:45099)
0926 23:23:11.034224 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.034241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.034411 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.034761 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.034765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.035231 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:11.035517 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.035526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.036516 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.036522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.038078 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.038117 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.039286 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.039290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.039444 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.040253 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:11.040274 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:11.041603 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:11.045505 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:11.045517 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:11.045521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:11.045929 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:11.046309 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:11.046314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:11.046318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:11.046449 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:11.046964 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:11.046969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:11.047152 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.047293 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.047657 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":363,"threads_started":1}
I20260926 23:23:11.048144 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.033226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45099 (local address 127.0.97.62:33425)
0926 23:23:11.033439 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.033444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.034335 (+ 891us) server_negotiation.cc:407] Connection header received
0926 23:23:11.034553 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.034562 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.034634 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.034737 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:11.035665 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.036365 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.038306 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.038978 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.039056 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.041816 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:11.041821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:11.041824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:11.041882 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:11.045364 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.046052 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.046059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.046063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.046190 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.046557 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.046563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.046567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.046844 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:11.046994 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.047409 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.047656 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:23:11.058269 1865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37285
I20260926 23:23:11.058359 1999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37285 every 8 connection(s)
I20260926 23:23:11.059559 1865 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:11.061686 2000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33425
I20260926 23:23:11.061930 2000 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:11.062615 2000 heartbeater.cc:511] Master 127.0.97.62:33425 requested a full tablet report, sending...
I20260926 23:23:11.064055 1514 ts_manager.cc:194] Registered new tserver with Master: 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3:37285)
I20260926 23:23:11.064996 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33435
I20260926 23:23:11.068749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 1865
I20260926 23:23:11.068894 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:11.069252 388 external_mini_cluster.cc:1428] Setting key dee19a8f92a64f1e741a98d933b455e7
I20260926 23:23:11.071908 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:23:11 dist-test-slave-q5g8 krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:23:11 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:11.120586 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.104263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.0.97.62:33425)
0926 23:23:11.104485 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.104494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.105076 (+ 582us) server_negotiation.cc:407] Connection header received
0926 23:23:11.105270 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.105275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.105354 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.105452 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:11.106877 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.107642 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.109055 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.109475 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.109599 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.113982 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:11.113989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:11.113992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:11.114035 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:11.117291 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.118238 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.118244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.118248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.118453 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.119017 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.119022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.119024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.119328 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:11.119538 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.119895 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.120140 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:23:11.126660 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:11.134197 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:11.166013 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.148017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.0.97.2:36225)
0926 23:23:11.154512 (+ 6495us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.154523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.155110 (+ 587us) server_negotiation.cc:407] Connection header received
0926 23:23:11.155558 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.155571 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.155909 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.157350 (+ 1441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.157578 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.159222 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.162552 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.164198 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.164338 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.164598 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.165461 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.165654 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6314,"thread_start_us":6169,"threads_started":1}
I20260926 23:23:11.166105 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.149268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34455 (local address 127.0.0.1:57800)
0926 23:23:11.156596 (+ 7328us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.156619 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.156765 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.157966 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.157971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.157999 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.158393 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.158406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.160613 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.160632 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.163068 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.163083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.165017 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.165024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.165234 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.165272 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.165361 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.165597 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7206,"mutex_wait_us":5275,"thread_start_us":1786,"threads_started":1}
I20260926 23:23:11.166450 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.149751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.97.1:34455)
0926 23:23:11.157273 (+ 7522us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.157284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.157326 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:23:11.157438 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.157447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.157725 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.158628 (+ 903us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.158826 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.160420 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.163254 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.164750 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.165574 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.165818 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.165892 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.166076 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7316,"thread_start_us":8245,"threads_started":1}
I20260926 23:23:11.167309 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.147925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36225 (local address 127.0.0.1:57024)
0926 23:23:11.154901 (+ 6976us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.154964 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.155260 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.156493 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.156502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.156538 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.157095 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.157118 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.159456 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.159466 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.162347 (+ 2881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.162366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.164842 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.164849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.165145 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.165211 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.166741 (+ 1530us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.166918 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6556,"mutex_wait_us":57,"thread_start_us":443,"threads_started":1}
I20260926 23:23:11.168835 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.148408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37285 (local address 127.0.0.1:52894)
0926 23:23:11.155903 (+ 7495us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.155923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.156047 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.160677 (+ 4630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.160727 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.160764 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.161491 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.161504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.163418 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.163424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.165764 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.165779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.167864 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.167872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.168059 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.168097 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.168200 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.168360 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7365,"mutex_wait_us":1,"thread_start_us":7665,"threads_started":1}
I20260926 23:23:11.171134 1791 tablet_service.cc:1467] Processing CreateTablet for tablet b1a70b0c3ba84b1a9323b8424d5d711f (DEFAULT_TABLE table=test-table [id=cfa0afb8302e4508b4dc8c5c0f149f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:11.171137 1652 tablet_service.cc:1467] Processing CreateTablet for tablet b1a70b0c3ba84b1a9323b8424d5d711f (DEFAULT_TABLE table=test-table [id=cfa0afb8302e4508b4dc8c5c0f149f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:11.172796 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a70b0c3ba84b1a9323b8424d5d711f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.172816 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a70b0c3ba84b1a9323b8424d5d711f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.173494 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.149043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.97.3:37285)
0926 23:23:11.160025 (+ 10982us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.160034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.160068 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:23:11.160167 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.160175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.160434 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.161568 (+ 1134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.161928 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.163252 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.165967 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.167639 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.167792 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.168035 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.168298 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.173188 (+ 4890us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10800,"thread_start_us":10616,"threads_started":1}
I20260926 23:23:11.177546 1930 tablet_service.cc:1467] Processing CreateTablet for tablet b1a70b0c3ba84b1a9323b8424d5d711f (DEFAULT_TABLE table=test-table [id=cfa0afb8302e4508b4dc8c5c0f149f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:11.179047 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a70b0c3ba84b1a9323b8424d5d711f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.190064 2019 tablet_bootstrap.cc:492] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Bootstrap starting.
I20260926 23:23:11.190956 2020 tablet_bootstrap.cc:492] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Bootstrap starting.
I20260926 23:23:11.192241 2021 tablet_bootstrap.cc:492] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Bootstrap starting.
I20260926 23:23:11.195372 2021 tablet_bootstrap.cc:654] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.195542 2020 tablet_bootstrap.cc:654] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.195990 2019 tablet_bootstrap.cc:654] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.196643 2021 log.cc:824] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:11.196987 2020 log.cc:824] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:11.197280 2019 log.cc:824] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:11.203541 2019 tablet_bootstrap.cc:492] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: No bootstrap required, opened a new log
I20260926 23:23:11.203881 2019 ts_tablet_manager.cc:1397] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260926 23:23:11.204166 2020 tablet_bootstrap.cc:492] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: No bootstrap required, opened a new log
I20260926 23:23:11.204353 2021 tablet_bootstrap.cc:492] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: No bootstrap required, opened a new log
I20260926 23:23:11.204502 2020 ts_tablet_manager.cc:1397] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260926 23:23:11.204680 2021 ts_tablet_manager.cc:1397] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 23:23:11.213135 2021 raft_consensus.cc:348] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.213661 2021 raft_consensus.cc:374] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.213820 2021 raft_consensus.cc:729] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb61606105a45d3ba3a4cdfae8ec765, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.214673 2021 consensus_queue.cc:260] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.216038 2000 heartbeater.cc:503] Master 127.0.97.62:33425 was elected leader, sending a full tablet report...
I20260926 23:23:11.216248 2019 raft_consensus.cc:348] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.216267 2020 raft_consensus.cc:348] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.216769 2019 raft_consensus.cc:374] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.216750 2020 raft_consensus.cc:374] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.216871 2019 raft_consensus.cc:729] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3d423503db49ca9ff71314bd9be3dd, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.216867 2020 raft_consensus.cc:729] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5ed0ec76ac46cf8879afa30108d38f, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.216897 2021 ts_tablet_manager.cc:1428] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260926 23:23:11.217600 2019 consensus_queue.cc:260] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.217602 2020 consensus_queue.cc:260] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.218775 1861 heartbeater.cc:503] Master 127.0.97.62:33425 was elected leader, sending a full tablet report...
I20260926 23:23:11.219558 2019 ts_tablet_manager.cc:1428] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
I20260926 23:23:11.219650 2027 raft_consensus.cc:482] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:11.219900 2027 raft_consensus.cc:504] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.221372 2020 ts_tablet_manager.cc:1428] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260926 23:23:11.226743 2027 leader_election.cc:288] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2:36225), 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3:37285)
I20260926 23:23:11.232118 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.222172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36225 (local address 127.0.97.1:53727)
0926 23:23:11.224714 (+ 2542us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.224737 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.224934 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.225507 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.225512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.225539 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.226065 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.226082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.227605 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.227611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.229992 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.230004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.231506 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.231511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.231635 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.231660 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.231740 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.231856 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2370,"thread_start_us":2198,"threads_started":1}
I20260926 23:23:11.232239 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.223188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53727 (local address 127.0.97.2:36225)
0926 23:23:11.223415 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.223421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.224819 (+ 1398us) server_negotiation.cc:407] Connection header received
0926 23:23:11.225107 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.225112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.225190 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.225259 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.226303 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.227454 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.230258 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.231284 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.231441 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.231565 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.231803 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.231942 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:23:11.233453 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.225928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42297 (local address 127.0.97.3:37285)
0926 23:23:11.226122 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.226129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.226846 (+ 717us) server_negotiation.cc:407] Connection header received
0926 23:23:11.227148 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.227154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.227231 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.227338 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.228121 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.229280 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.231097 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.232365 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.232520 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.232636 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.233058 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.233192 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:23:11.233299 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a70b0c3ba84b1a9323b8424d5d711f" candidate_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d423503db49ca9ff71314bd9be3dd" is_pre_election: true
I20260926 23:23:11.233582 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.226003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37285 (local address 127.0.97.1:42297)
0926 23:23:11.226742 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.226762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.226944 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.227377 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.227382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.227407 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.227959 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.227973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.229416 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.229421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.230932 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.230944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.232722 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.232733 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.232889 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.232914 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.233032 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.233175 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":402,"threads_started":1}
I20260926 23:23:11.233867 1812 raft_consensus.cc:2454] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5ed0ec76ac46cf8879afa30108d38f in term 0.
I20260926 23:23:11.234629 1605 leader_election.cc:302] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5ed0ec76ac46cf8879afa30108d38f, 8e3d423503db49ca9ff71314bd9be3dd; no voters:
I20260926 23:23:11.234716 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a70b0c3ba84b1a9323b8424d5d711f" candidate_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" is_pre_election: true
I20260926 23:23:11.235442 2027 raft_consensus.cc:2781] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:11.235457 1955 raft_consensus.cc:2454] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5ed0ec76ac46cf8879afa30108d38f in term 0.
I20260926 23:23:11.235574 2027 raft_consensus.cc:482] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:11.235708 2027 raft_consensus.cc:3037] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.238247 2027 raft_consensus.cc:504] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.238970 2027 leader_election.cc:288] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [CANDIDATE]: Term 1 election: Requested vote from peers 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2:36225), 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3:37285)
I20260926 23:23:11.239434 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a70b0c3ba84b1a9323b8424d5d711f" candidate_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d423503db49ca9ff71314bd9be3dd"
I20260926 23:23:11.239543 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a70b0c3ba84b1a9323b8424d5d711f" candidate_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb61606105a45d3ba3a4cdfae8ec765"
I20260926 23:23:11.239684 1812 raft_consensus.cc:3037] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.239773 1955 raft_consensus.cc:3037] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.242125 1955 raft_consensus.cc:2454] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5ed0ec76ac46cf8879afa30108d38f in term 1.
I20260926 23:23:11.242380 1812 raft_consensus.cc:2454] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5ed0ec76ac46cf8879afa30108d38f in term 1.
I20260926 23:23:11.242617 1602 leader_election.cc:302] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb61606105a45d3ba3a4cdfae8ec765, 4b5ed0ec76ac46cf8879afa30108d38f; no voters:
I20260926 23:23:11.242938 2027 raft_consensus.cc:2781] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:11.243412 2027 raft_consensus.cc:686] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 LEADER]: Becoming Leader. State: Replica: 4b5ed0ec76ac46cf8879afa30108d38f, State: Running, Role: LEADER
I20260926 23:23:11.244055 2027 consensus_queue.cc:237] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.248908 1515 catalog_manager.cc:5495] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } health_report { overall_health: UNKNOWN } } }
W20260926 23:23:11.274413 1723 tablet.cc:2369] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:11.277221 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.271222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.0.97.1:34455)
0926 23:23:11.271444 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.271450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.271472 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:23:11.271776 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.271781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.271866 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.272016 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:11.272370 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.273121 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.274659 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.275257 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.275374 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.275969 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.276481 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.276601 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.276839 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.276934 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:23:11.285519 1955 raft_consensus.cc:1261] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Refusing update from remote peer 4b5ed0ec76ac46cf8879afa30108d38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:11.285522 1812 raft_consensus.cc:1261] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Refusing update from remote peer 4b5ed0ec76ac46cf8879afa30108d38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:11.286715 2030 consensus_queue.cc:1035] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:11.287119 2027 consensus_queue.cc:1035] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:23:11.291455 1862 tablet.cc:2369] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:11.301115 2035 mvcc.cc:205] Tried to move back new op lower bound from 7333744604295155712 to 7333744604140560384. Current Snapshot: MvccSnapshot[applied={T|T < 7333744604295155712}]
I20260926 23:23:11.304011 2034 mvcc.cc:205] Tried to move back new op lower bound from 7333744604295155712 to 7333744604140560384. Current Snapshot: MvccSnapshot[applied={T|T < 7333744604295155712}]
I20260926 23:23:11.320441 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:11.320884 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:11.325372 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a: Sending DeleteTablet for 3 replicas of tablet b1a70b0c3ba84b1a9323b8424d5d711f
I20260926 23:23:11.326658 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a70b0c3ba84b1a9323b8424d5d711f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57024
I20260926 23:23:11.326771 1930 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a70b0c3ba84b1a9323b8424d5d711f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52894
I20260926 23:23:11.326999 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a70b0c3ba84b1a9323b8424d5d711f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57800
I20260926 23:23:11.335734 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:11.337097 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:11.337289 2043 tablet_replica.cc:330] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: stopping tablet replica
I20260926 23:23:11.338457 2043 raft_consensus.cc:2229] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:11.339550 2043 raft_consensus.cc:2258] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:11.340960 2042 tablet_replica.cc:330] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: stopping tablet replica
I20260926 23:23:11.341226 2044 tablet_replica.cc:330] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: stopping tablet replica
I20260926 23:23:11.341549 2042 raft_consensus.cc:2229] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:11.342131 2042 raft_consensus.cc:2258] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:11.343560 2043 ts_tablet_manager.cc:1905] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:11.344377 2044 raft_consensus.cc:2229] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:11.345816 2042 ts_tablet_manager.cc:1905] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:11.346408 1930 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecd41f192514dbb8c44e3a32087169a (DEFAULT_TABLE table=test-table [id=8f0bacddd96d441399c7850690362456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:11.346501 2044 raft_consensus.cc:2258] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:11.347026 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ecd41f192514dbb8c44e3a32087169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.348733 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecd41f192514dbb8c44e3a32087169a (DEFAULT_TABLE table=test-table [id=8f0bacddd96d441399c7850690362456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:11.349298 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ecd41f192514dbb8c44e3a32087169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.353117 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecd41f192514dbb8c44e3a32087169a (DEFAULT_TABLE table=test-table [id=8f0bacddd96d441399c7850690362456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:11.354357 2042 ts_tablet_manager.cc:1918] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:11.354655 2042 log.cc:1196] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/b1a70b0c3ba84b1a9323b8424d5d711f
I20260926 23:23:11.355297 2042 ts_tablet_manager.cc:1939] T b1a70b0c3ba84b1a9323b8424d5d711f P 8e3d423503db49ca9ff71314bd9be3dd: Deleting consensus metadata
I20260926 23:23:11.356490 2044 ts_tablet_manager.cc:1905] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:11.357250 1502 catalog_manager.cc:4841] TS 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2:36225): tablet b1a70b0c3ba84b1a9323b8424d5d711f (table test-table [id=cfa0afb8302e4508b4dc8c5c0f149f84]) successfully deleted
I20260926 23:23:11.357882 2021 tablet_bootstrap.cc:492] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Bootstrap starting.
I20260926 23:23:11.361106 2021 tablet_bootstrap.cc:654] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.364029 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ecd41f192514dbb8c44e3a32087169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.367268 2019 tablet_bootstrap.cc:492] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: Bootstrap starting.
I20260926 23:23:11.367347 2021 tablet_bootstrap.cc:492] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: No bootstrap required, opened a new log
I20260926 23:23:11.367509 2021 ts_tablet_manager.cc:1397] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 23:23:11.367777 2044 ts_tablet_manager.cc:1918] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:11.367960 2043 ts_tablet_manager.cc:1918] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:11.368052 2044 log.cc:1196] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/b1a70b0c3ba84b1a9323b8424d5d711f
I20260926 23:23:11.368274 2043 log.cc:1196] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/b1a70b0c3ba84b1a9323b8424d5d711f
I20260926 23:23:11.368513 2021 raft_consensus.cc:348] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.368937 2021 raft_consensus.cc:374] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.369112 2019 tablet_bootstrap.cc:654] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.369138 2021 raft_consensus.cc:729] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb61606105a45d3ba3a4cdfae8ec765, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.369020 2043 ts_tablet_manager.cc:1939] T b1a70b0c3ba84b1a9323b8424d5d711f P 2cb61606105a45d3ba3a4cdfae8ec765: Deleting consensus metadata
I20260926 23:23:11.369515 2021 consensus_queue.cc:260] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.370339 2021 ts_tablet_manager.cc:1428] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:11.370870 2044 ts_tablet_manager.cc:1939] T b1a70b0c3ba84b1a9323b8424d5d711f P 4b5ed0ec76ac46cf8879afa30108d38f: Deleting consensus metadata
I20260926 23:23:11.372079 2019 tablet_bootstrap.cc:492] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: No bootstrap required, opened a new log
I20260926 23:23:11.372295 2019 ts_tablet_manager.cc:1397] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 23:23:11.373257 2019 raft_consensus.cc:348] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.373607 2019 raft_consensus.cc:374] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.373780 2019 raft_consensus.cc:729] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3d423503db49ca9ff71314bd9be3dd, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.374063 1499 catalog_manager.cc:4841] TS 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3:37285): tablet b1a70b0c3ba84b1a9323b8424d5d711f (table test-table [id=cfa0afb8302e4508b4dc8c5c0f149f84]) successfully deleted
I20260926 23:23:11.375106 1502 catalog_manager.cc:4841] TS 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1:34455): tablet b1a70b0c3ba84b1a9323b8424d5d711f (table test-table [id=cfa0afb8302e4508b4dc8c5c0f149f84]) successfully deleted
I20260926 23:23:11.374747 2019 consensus_queue.cc:260] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.376001 2019 ts_tablet_manager.cc:1428] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 23:23:11.381256 2020 tablet_bootstrap.cc:492] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: Bootstrap starting.
I20260926 23:23:11.382589 2020 tablet_bootstrap.cc:654] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.385133 2020 tablet_bootstrap.cc:492] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: No bootstrap required, opened a new log
I20260926 23:23:11.385301 2020 ts_tablet_manager.cc:1397] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260926 23:23:11.386109 2020 raft_consensus.cc:348] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.386281 2020 raft_consensus.cc:374] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.386405 2020 raft_consensus.cc:729] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5ed0ec76ac46cf8879afa30108d38f, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.386662 2020 consensus_queue.cc:260] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.387251 2020 ts_tablet_manager.cc:1428] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:11.458181 2026 raft_consensus.cc:482] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:11.458446 2026 raft_consensus.cc:504] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.460150 2026 leader_election.cc:288] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1:34455), 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3:37285)
I20260926 23:23:11.468837 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.461381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60873 (local address 127.0.97.1:34455)
0926 23:23:11.461582 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.461588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.461611 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:23:11.461683 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.461689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.461811 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.461904 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.462616 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.464476 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.466728 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.467519 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.467678 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.467789 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.468257 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.468423 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:23:11.469105 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.461006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46247 (local address 127.0.97.3:37285)
0926 23:23:11.461190 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.461197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.461800 (+ 603us) server_negotiation.cc:407] Connection header received
0926 23:23:11.461970 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.461977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.462061 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.462174 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.462821 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.463772 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.466208 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.467578 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.467718 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.467838 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.468346 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.468524 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:23:11.469472 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.460339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34455 (local address 127.0.97.2:60873)
0926 23:23:11.461056 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.461083 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.461251 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.461972 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.461977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.462011 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.462423 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.462435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.464669 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.464698 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.466580 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.466589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.467748 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.467753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.468059 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.468095 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.468202 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.468578 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":48,"thread_start_us":407,"threads_started":1}
I20260926 23:23:11.469866 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.460507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37285 (local address 127.0.97.2:46247)
0926 23:23:11.461681 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.461697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.461816 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.462194 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.462201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.462231 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.462670 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.462678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.463898 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.463903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.466033 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.466047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.467949 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.467955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.468165 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.468206 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.469285 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.469452 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":27,"thread_start_us":924,"threads_started":1}
I20260926 23:23:11.470438 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecd41f192514dbb8c44e3a32087169a" candidate_uuid: "8e3d423503db49ca9ff71314bd9be3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" is_pre_election: true
I20260926 23:23:11.470624 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecd41f192514dbb8c44e3a32087169a" candidate_uuid: "8e3d423503db49ca9ff71314bd9be3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" is_pre_election: true
I20260926 23:23:11.470938 1954 raft_consensus.cc:2454] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e3d423503db49ca9ff71314bd9be3dd in term 0.
I20260926 23:23:11.470922 1672 raft_consensus.cc:2454] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e3d423503db49ca9ff71314bd9be3dd in term 0.
I20260926 23:23:11.471473 1742 leader_election.cc:302] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb61606105a45d3ba3a4cdfae8ec765, 8e3d423503db49ca9ff71314bd9be3dd; no voters:
I20260926 23:23:11.471911 2026 raft_consensus.cc:2781] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:11.472051 2026 raft_consensus.cc:482] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:11.472149 2026 raft_consensus.cc:3037] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.476158 2026 raft_consensus.cc:504] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.476982 2026 leader_election.cc:288] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [CANDIDATE]: Term 1 election: Requested vote from peers 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1:34455), 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3:37285)
I20260926 23:23:11.477447 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecd41f192514dbb8c44e3a32087169a" candidate_uuid: "8e3d423503db49ca9ff71314bd9be3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5ed0ec76ac46cf8879afa30108d38f"
I20260926 23:23:11.477460 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecd41f192514dbb8c44e3a32087169a" candidate_uuid: "8e3d423503db49ca9ff71314bd9be3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb61606105a45d3ba3a4cdfae8ec765"
I20260926 23:23:11.477664 1672 raft_consensus.cc:3037] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.477695 1954 raft_consensus.cc:3037] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.480350 1672 raft_consensus.cc:2454] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e3d423503db49ca9ff71314bd9be3dd in term 1.
I20260926 23:23:11.480831 1745 leader_election.cc:302] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5ed0ec76ac46cf8879afa30108d38f, 8e3d423503db49ca9ff71314bd9be3dd; no voters:
I20260926 23:23:11.480966 1954 raft_consensus.cc:2454] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e3d423503db49ca9ff71314bd9be3dd in term 1.
I20260926 23:23:11.481161 2026 raft_consensus.cc:2781] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:11.481709 2026 raft_consensus.cc:686] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 1 LEADER]: Becoming Leader. State: Replica: 8e3d423503db49ca9ff71314bd9be3dd, State: Running, Role: LEADER
I20260926 23:23:11.482124 2026 consensus_queue.cc:237] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.485714 1513 catalog_manager.cc:5495] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8e3d423503db49ca9ff71314bd9be3dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } health_report { overall_health: UNKNOWN } } }
Sep 26 23:23:11 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:11.554675 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.535509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33425 (local address 127.0.0.1:43710)
0926 23:23:11.536246 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.536266 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.536438 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.537011 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.537017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.537934 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:11.538314 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.538326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.539935 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.539941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.541533 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.541547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.542599 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.542606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.542778 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.543569 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:11.543595 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:11.547070 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:11.551488 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:11.551503 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:11.551512 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:11.552166 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:11.552606 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:11.552611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:11.552615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:11.552829 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:11.553495 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:11.553503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:11.553691 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.553865 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.554181 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":386,"threads_started":1}
I20260926 23:23:11.555346 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.535420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.0.97.62:33425)
0926 23:23:11.535588 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.535594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.536378 (+ 784us) server_negotiation.cc:407] Connection header received
0926 23:23:11.536591 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.536598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.536698 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.536777 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:11.538507 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.539785 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.541687 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.542405 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.542549 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.547277 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:11.547285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:11.547291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:11.547332 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:11.551322 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.552311 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.552317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.552322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.552486 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.552945 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.552951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.552955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.553350 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:11.553506 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.554018 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.554368 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":54}
I20260926 23:23:11.561879 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43710:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:23:11.562361 2001 tablet.cc:2369] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:23:11.563236 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:11.571308 1791 tablet_service.cc:1467] Processing CreateTablet for tablet c276a24018b443d5a666a55129a5bd90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e56e706126974a078d0c2b5eeccd01db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:23:11.571355 1652 tablet_service.cc:1467] Processing CreateTablet for tablet c276a24018b443d5a666a55129a5bd90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e56e706126974a078d0c2b5eeccd01db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:23:11.571918 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c276a24018b443d5a666a55129a5bd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.571973 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c276a24018b443d5a666a55129a5bd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.573252 1930 tablet_service.cc:1467] Processing CreateTablet for tablet c276a24018b443d5a666a55129a5bd90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e56e706126974a078d0c2b5eeccd01db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:23:11.573823 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c276a24018b443d5a666a55129a5bd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:11.578099 2019 tablet_bootstrap.cc:492] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: Bootstrap starting.
I20260926 23:23:11.579587 2019 tablet_bootstrap.cc:654] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.581885 2019 tablet_bootstrap.cc:492] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: No bootstrap required, opened a new log
I20260926 23:23:11.582072 2019 ts_tablet_manager.cc:1397] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 23:23:11.582654 2019 raft_consensus.cc:348] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.582881 2019 raft_consensus.cc:374] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.582985 2019 raft_consensus.cc:729] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3d423503db49ca9ff71314bd9be3dd, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.583251 2019 consensus_queue.cc:260] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.583683 2026 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.583880 2019 ts_tablet_manager.cc:1428] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:11.584007 2026 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:11.586200 2021 tablet_bootstrap.cc:492] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: Bootstrap starting.
I20260926 23:23:11.588415 2021 tablet_bootstrap.cc:654] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.590297 2020 tablet_bootstrap.cc:492] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: Bootstrap starting.
I20260926 23:23:11.590622 2021 tablet_bootstrap.cc:492] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: No bootstrap required, opened a new log
I20260926 23:23:11.590739 2021 ts_tablet_manager.cc:1397] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260926 23:23:11.591240 2021 raft_consensus.cc:348] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.591408 2021 raft_consensus.cc:374] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.591454 2021 raft_consensus.cc:729] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb61606105a45d3ba3a4cdfae8ec765, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.591681 2021 consensus_queue.cc:260] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.592183 2021 ts_tablet_manager.cc:1428] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 23:23:11.592196 2020 tablet_bootstrap.cc:654] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:11.592077 2025 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.592373 2025 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:11.595901 2020 tablet_bootstrap.cc:492] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: No bootstrap required, opened a new log
I20260926 23:23:11.596098 2020 ts_tablet_manager.cc:1397] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 23:23:11.596917 2020 raft_consensus.cc:348] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.597180 2020 raft_consensus.cc:374] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:11.597342 2020 raft_consensus.cc:729] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5ed0ec76ac46cf8879afa30108d38f, State: Initialized, Role: FOLLOWER
I20260926 23:23:11.597740 2020 consensus_queue.cc:260] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.598255 2030 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.598562 2030 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:11.599148 2020 ts_tablet_manager.cc:1428] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:11.606266 2025 raft_consensus.cc:482] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:11.606500 2025 raft_consensus.cc:504] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.608408 2025 leader_election.cc:288] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1:34455), 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2:36225)
I20260926 23:23:11.617165 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.608723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51609 (local address 127.0.97.1:34455)
0926 23:23:11.608956 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.608963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.609378 (+ 415us) server_negotiation.cc:407] Connection header received
0926 23:23:11.609698 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.609704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.609830 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.609927 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.610964 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.612213 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.614974 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.616197 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.616362 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.616472 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.616743 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.616883 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:23:11.617259 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.608623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34455 (local address 127.0.97.3:51609)
0926 23:23:11.609279 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.609300 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.609483 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.610388 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.610396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.610431 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.610759 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.610785 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.612363 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.612368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.614810 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.614822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.616377 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.616383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.616561 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.616585 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.616692 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.616853 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":3,"thread_start_us":360,"threads_started":1}
I20260926 23:23:11.618271 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276a24018b443d5a666a55129a5bd90" candidate_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" is_pre_election: true
I20260926 23:23:11.618566 1672 raft_consensus.cc:2454] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb61606105a45d3ba3a4cdfae8ec765 in term 0.
I20260926 23:23:11.619268 1884 leader_election.cc:302] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb61606105a45d3ba3a4cdfae8ec765, 4b5ed0ec76ac46cf8879afa30108d38f; no voters:
I20260926 23:23:11.619736 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.609735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38427 (local address 127.0.97.2:36225)
0926 23:23:11.610253 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.610262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.611429 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 23:23:11.611607 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.611610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.611681 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.611783 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:11.612235 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.613820 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.616528 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.618003 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.618111 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.618218 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.619317 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.619461 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
I20260926 23:23:11.619805 2025 raft_consensus.cc:2781] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:11.619946 2025 raft_consensus.cc:482] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:11.620039 2025 raft_consensus.cc:3037] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.623554 2025 raft_consensus.cc:504] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.624807 2025 leader_election.cc:288] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [CANDIDATE]: Term 1 election: Requested vote from peers 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1:34455), 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2:36225)
I20260926 23:23:11.625075 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.609396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36225 (local address 127.0.97.3:38427)
0926 23:23:11.611328 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.611342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.611469 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.611799 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.611803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.611825 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:11.612080 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.612087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.613961 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.613967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.616327 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.616345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.618418 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.618425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.619116 (+ 691us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.619160 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.619281 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.619440 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1841,"thread_start_us":1681,"threads_started":1}
I20260926 23:23:11.625344 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276a24018b443d5a666a55129a5bd90" candidate_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5ed0ec76ac46cf8879afa30108d38f"
I20260926 23:23:11.625583 1672 raft_consensus.cc:3037] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.627779 1672 raft_consensus.cc:2454] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb61606105a45d3ba3a4cdfae8ec765 in term 1.
I20260926 23:23:11.628336 1884 leader_election.cc:302] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb61606105a45d3ba3a4cdfae8ec765, 4b5ed0ec76ac46cf8879afa30108d38f; no voters:
I20260926 23:23:11.628666 2025 raft_consensus.cc:2781] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:11.628896 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276a24018b443d5a666a55129a5bd90" candidate_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d423503db49ca9ff71314bd9be3dd" is_pre_election: true
I20260926 23:23:11.629156 1812 raft_consensus.cc:2454] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb61606105a45d3ba3a4cdfae8ec765 in term 0.
I20260926 23:23:11.629499 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276a24018b443d5a666a55129a5bd90" candidate_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d423503db49ca9ff71314bd9be3dd"
I20260926 23:23:11.629757 1810 raft_consensus.cc:3037] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:11.632303 1810 raft_consensus.cc:2454] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb61606105a45d3ba3a4cdfae8ec765 in term 1.
I20260926 23:23:11.633147 2025 raft_consensus.cc:686] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb61606105a45d3ba3a4cdfae8ec765, State: Running, Role: LEADER
I20260926 23:23:11.633674 2025 consensus_queue.cc:237] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } }
I20260926 23:23:11.636056 2063 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: TxnStatusTablet state changed. Reason: New leader 2cb61606105a45d3ba3a4cdfae8ec765. Latest consensus state: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.636451 2063 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:23:11.637332 2065 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:23:11.637552 2065 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:23:11.638098 1513 catalog_manager.cc:5495] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } health_report { overall_health: HEALTHY } } }
I20260926 23:23:11.659606 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.651283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37285 (local address 127.0.0.1:52898)
0926 23:23:11.651703 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.651725 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.651885 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.652478 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.652483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.652497 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:11.652748 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.652760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.654379 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.654385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.656280 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.656296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.657501 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.657508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.657716 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.657792 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.658978 (+ 1186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.659003 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.659099 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.659261 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260926 23:23:11.659592 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.651311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.0.97.3:37285)
0926 23:23:11.651534 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.651541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.651818 (+ 277us) server_negotiation.cc:407] Connection header received
0926 23:23:11.652097 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.652102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.652229 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.652410 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:11.652914 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.654212 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.656477 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.657307 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.657461 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.658031 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.658843 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.659022 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.659126 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.659224 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:23:11.665122 1810 raft_consensus.cc:1261] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Refusing update from remote peer 2cb61606105a45d3ba3a4cdfae8ec765: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:11.666157 1672 raft_consensus.cc:1261] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Refusing update from remote peer 2cb61606105a45d3ba3a4cdfae8ec765: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:11.667390 2025 consensus_queue.cc:1035] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:11.667979 2063 consensus_queue.cc:1035] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:11.683060 2026 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: TxnStatusTablet state changed. Reason: New leader 2cb61606105a45d3ba3a4cdfae8ec765. Latest consensus state: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.683418 2026 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:11.686151 2030 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: TxnStatusTablet state changed. Reason: New leader 2cb61606105a45d3ba3a4cdfae8ec765. Latest consensus state: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.686940 2030 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:11.690663 2033 mvcc.cc:205] Tried to move back new op lower bound from 7333744605849890816 to 7333744605736611840. Current Snapshot: MvccSnapshot[applied={T|T < 7333744605849890816}]
I20260926 23:23:11.691711 2030 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.692003 2030 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 4b5ed0ec76ac46cf8879afa30108d38f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:11.695021 2025 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.695416 2025 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:23:11.699759 2026 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.700074 2026 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 8e3d423503db49ca9ff71314bd9be3dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:11.702229 2063 tablet_replica.cc:439] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 } } peers { permanent_uuid: "8e3d423503db49ca9ff71314bd9be3dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36225 } } peers { permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 } } }
I20260926 23:23:11.702500 2063 tablet_replica.cc:442] T c276a24018b443d5a666a55129a5bd90 P 2cb61606105a45d3ba3a4cdfae8ec765: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:23:11.715457 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.706993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.97.2:36225)
0926 23:23:11.707146 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.707150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.707538 (+ 388us) server_negotiation.cc:407] Connection header received
0926 23:23:11.707651 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.707656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.707803 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.707980 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:11.708605 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.709875 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.712099 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.712924 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.713062 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.713805 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.714682 (+ 877us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.714933 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.714997 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.715101 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 23:23:11.738961 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.730460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42617 (local address 127.0.97.3:37285)
0926 23:23:11.730917 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.730924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.731322 (+ 398us) server_negotiation.cc:407] Connection header received
0926 23:23:11.731559 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.731566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.731672 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.731804 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:11.732887 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.734218 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.736300 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.737249 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.737406 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.737950 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.738143 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.738256 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.738478 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.738576 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260926 23:23:11.738994 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.730495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37285 (local address 127.0.97.2:42617)
0926 23:23:11.731193 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.731216 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.731387 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.731831 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.731836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.731850 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:11.732243 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.732257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.734380 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.734387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.736130 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.736145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.737453 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.737460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.737640 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.737723 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.738301 (+ 578us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.738317 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.738416 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.738585 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":437,"threads_started":1}
I20260926 23:23:11.753381 1954 raft_consensus.cc:1261] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Refusing update from remote peer 8e3d423503db49ca9ff71314bd9be3dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:11.754637 1672 raft_consensus.cc:1261] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Refusing update from remote peer 8e3d423503db49ca9ff71314bd9be3dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:11.754617 2050 consensus_queue.cc:1035] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb61606105a45d3ba3a4cdfae8ec765" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:11.755662 2050 consensus_queue.cc:1035] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5ed0ec76ac46cf8879afa30108d38f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:11.828516 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.817559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32963 (local address 127.0.97.2:36225)
0926 23:23:11.818120 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.818126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.820086 (+ 1960us) server_negotiation.cc:407] Connection header received
0926 23:23:11.820278 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.820285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.820377 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.820481 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:11.821094 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.822201 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.826289 (+ 4088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.827044 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.827185 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.827736 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.827880 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.827973 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.828126 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.828219 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431}
I20260926 23:23:11.828992 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.817543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36225 (local address 127.0.97.3:32963)
0926 23:23:11.819958 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:11.819979 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:11.820129 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:11.820517 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:11.820523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:11.820535 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:11.820924 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.820937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.822361 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.822367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.826102 (+ 3735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:11.826124 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.827242 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.827249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:11.827450 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.827531 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:11.827988 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:11.828001 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 23:23:11.828087 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:23:11.828255 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2195,"thread_start_us":2059,"threads_started":1}
I20260926 23:23:11.884128 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:11.884392 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:11.887970 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f2711fdac6e41edaec5d683bbb8fc8a: Sending DeleteTablet for 3 replicas of tablet 0ecd41f192514dbb8c44e3a32087169a
I20260926 23:23:11.889070 1930 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ecd41f192514dbb8c44e3a32087169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52894
I20260926 23:23:11.890110 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ecd41f192514dbb8c44e3a32087169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57800
I20260926 23:23:11.890332 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ecd41f192514dbb8c44e3a32087169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57024
I20260926 23:23:11.897080 2081 tablet_replica.cc:330] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: stopping tablet replica
I20260926 23:23:11.897444 2081 raft_consensus.cc:2229] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:11.897750 2081 raft_consensus.cc:2258] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:11.899111 2081 ts_tablet_manager.cc:1905] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:11.917055 2081 ts_tablet_manager.cc:1918] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:11.917343 2081 log.cc:1196] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/0ecd41f192514dbb8c44e3a32087169a
I20260926 23:23:11.917917 2081 ts_tablet_manager.cc:1939] T 0ecd41f192514dbb8c44e3a32087169a P 4b5ed0ec76ac46cf8879afa30108d38f: Deleting consensus metadata
I20260926 23:23:11.919258 1502 catalog_manager.cc:4841] TS 4b5ed0ec76ac46cf8879afa30108d38f (127.0.97.1:34455): tablet 0ecd41f192514dbb8c44e3a32087169a (table test-table [id=8f0bacddd96d441399c7850690362456]) successfully deleted
I20260926 23:23:11.924002 2082 tablet_replica.cc:330] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: stopping tablet replica
I20260926 23:23:11.925712 2082 raft_consensus.cc:2229] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:11.927103 2082 raft_consensus.cc:2258] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:11.927708 2080 tablet_replica.cc:330] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: stopping tablet replica
I20260926 23:23:11.929275 2080 raft_consensus.cc:2229] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:11.929590 2080 raft_consensus.cc:2258] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:11.929049 2082 ts_tablet_manager.cc:1905] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:11.931209 2080 ts_tablet_manager.cc:1905] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:23:11 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:11.957998 2082 ts_tablet_manager.cc:1918] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:11.960249 2080 ts_tablet_manager.cc:1918] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:11.960485 2082 log.cc:1196] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/0ecd41f192514dbb8c44e3a32087169a
I20260926 23:23:11.961023 2082 ts_tablet_manager.cc:1939] T 0ecd41f192514dbb8c44e3a32087169a P 8e3d423503db49ca9ff71314bd9be3dd: Deleting consensus metadata
I20260926 23:23:11.962312 1502 catalog_manager.cc:4841] TS 8e3d423503db49ca9ff71314bd9be3dd (127.0.97.2:36225): tablet 0ecd41f192514dbb8c44e3a32087169a (table test-table [id=8f0bacddd96d441399c7850690362456]) successfully deleted
I20260926 23:23:11.962622 2080 log.cc:1196] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/0ecd41f192514dbb8c44e3a32087169a
I20260926 23:23:11.963070 2080 ts_tablet_manager.cc:1939] T 0ecd41f192514dbb8c44e3a32087169a P 2cb61606105a45d3ba3a4cdfae8ec765: Deleting consensus metadata
I20260926 23:23:11.964023 1499 catalog_manager.cc:4841] TS 2cb61606105a45d3ba3a4cdfae8ec765 (127.0.97.3:37285): tablet 0ecd41f192514dbb8c44e3a32087169a (table test-table [id=8f0bacddd96d441399c7850690362456]) successfully deleted
I20260926 23:23:11.973460 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.926309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.0.97.1:34455)
0926 23:23:11.926491 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.926497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.937295 (+ 10798us) server_negotiation.cc:407] Connection header received
0926 23:23:11.938626 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.938632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.938708 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.938801 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:11.944718 (+ 5917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.945853 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.951752 (+ 5899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.952435 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.952535 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.965279 (+ 12744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:11.965291 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:11.965304 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:11.965402 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:11.968961 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.970050 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.970064 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.970068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.970241 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.970764 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.970770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.970782 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.971308 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:11.971601 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.972299 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.972589 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 23:23:11.976418 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57820
I20260926 23:23:11.994169 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.980459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.0.97.62:33425)
0926 23:23:11.980739 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.980749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:11.981184 (+ 435us) server_negotiation.cc:407] Connection header received
0926 23:23:11.981502 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:11.981510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:11.981614 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:11.981699 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:11.983640 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.984360 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.985690 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:11.986091 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:11.986235 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:11.988429 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:11.988437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:11.988446 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:11.988479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:11.991030 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.992109 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.992115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.992119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.992254 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:11.992604 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:11.992611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:11.992614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:11.992946 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:11.993285 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:11.993595 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 23:23:11.993809 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 23:23:11.994743 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43722
I20260926 23:23:11.996866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1586
I20260926 23:23:12.015825 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:11.998461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.0.97.1:34455)
0926 23:23:11.998628 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:11.998636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:12.003230 (+ 4594us) server_negotiation.cc:407] Connection header received
0926 23:23:12.003456 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:12.003462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:12.003541 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:12.003650 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:12.004810 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:12.005934 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:12.007116 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:12.007535 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:12.007671 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:12.010062 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:12.010068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:12.010073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:12.010113 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:12.012915 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:12.013541 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:12.013548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:12.013552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:12.013787 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:12.014256 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:12.014261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:12.014265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:12.014634 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:12.015059 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:12.015257 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 23:23:12.015467 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 23:23:12.016522 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57836
W20260926 23:23:12.017086 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:12.017230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 1586
I20260926 23:23:12.048204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1726
Sep 26 23:23:12 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:12.069343 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:12.049110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.0.97.2:36225)
0926 23:23:12.049315 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:12.049322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:12.049473 (+ 151us) server_negotiation.cc:407] Connection header received
0926 23:23:12.049658 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:12.049664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:12.049770 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:12.049893 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:12.051399 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:12.052543 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:12.054513 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:12.055213 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:12.055353 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:12.060793 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:12.060809 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:12.060826 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:12.060874 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:12.065220 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:12.066101 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:12.066110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:12.066115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:12.066308 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:12.066750 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:12.066755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:12.066759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:12.067205 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:12.067525 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:12.068712 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0926 23:23:12.068936 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 23:23:12.070387 1856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57046
W20260926 23:23:12.071138 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1726: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:12.071336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 1726
I20260926 23:23:12.089977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1865
Sep 26 23:23:12 dist-test-slave-q5g8 krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:12.106135 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:12.090771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.97.3:37285)
0926 23:23:12.090939 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:12.090945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:12.091086 (+ 141us) server_negotiation.cc:407] Connection header received
0926 23:23:12.091294 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:12.091301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:12.091365 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:12.091472 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:12.092873 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:12.093673 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:12.094928 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:12.095420 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:12.095554 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:12.099698 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:12.099712 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:12.099728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:12.099768 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:12.103138 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:12.103840 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:12.103849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:12.103852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:12.103994 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:12.104400 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:12.104405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:12.104408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:12.104837 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:12.105158 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:12.105431 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 23:23:12.105744 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 23:23:12.106938 1995 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52906
W20260926 23:23:12.107561 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:12.107720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 1865
I20260926 23:23:12.124238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 1483
I20260926 23:23:12.125633 1555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:12.244884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 1483
2026-09-26T23:23:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4595 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:12 dist-test-slave-q5g8 krb5kdc[2091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:12 dist-test-slave-q5g8 krb5kdc[2091](info): set up 2 sockets
Sep 26 23:23:12 dist-test-slave-q5g8 krb5kdc[2091](info): commencing operation
krb5kdc: starting...
W20260926 23:23:13.336943 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:13 dist-test-slave-q5g8 krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:13.521348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42589
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:13.830636 2104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:13.830956 2104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:13.831141 2104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:13.838284 2104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:13.838420 2104 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:23:13.838516 2104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:13.838634 2104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:13.838718 2104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:13.838805 2104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:13.848526 2104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42589
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:13.849864 2104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:13.851190 2104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:13.862854 2110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:13.863052 2112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:13.863386 2109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:13.863804 2104 server_base.cc:1034] running on GCE node
I20260926 23:23:13.864506 2104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:13.865830 2104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:13.867066 2104 hybrid_clock.cc:648] HybridClock initialized: now 1790464993866984 us; error 59 us; skew 500 ppm
I20260926 23:23:13.867388 2104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:13 dist-test-slave-q5g8 krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:13.874058 2104 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:13.875470 2104 webserver.cc:460] Webserver started at http://127.0.97.62:36113/ using document root <none> and password file <none>
I20260926 23:23:13.876151 2104 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:13.876288 2104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:13.876595 2104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:13.879437 2104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "64d3adafb1f74f388afc630d73a30170"
format_stamp: "Formatted at 2026-09-26 23:23:13 on dist-test-slave-q5g8"
server_key: "9a0280e6555a5804075f61e7ff8bc394"
server_key_iv: "5393bf675e9f6ee5a86d465292144825"
server_key_version: "encryptionkey@0"
I20260926 23:23:13.880177 2104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "64d3adafb1f74f388afc630d73a30170"
format_stamp: "Formatted at 2026-09-26 23:23:13 on dist-test-slave-q5g8"
server_key: "9a0280e6555a5804075f61e7ff8bc394"
server_key_iv: "5393bf675e9f6ee5a86d465292144825"
server_key_version: "encryptionkey@0"
I20260926 23:23:13.885187 2104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 23:23:13.888336 2119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:13.889667 2104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:13.889837 2104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "64d3adafb1f74f388afc630d73a30170"
format_stamp: "Formatted at 2026-09-26 23:23:13 on dist-test-slave-q5g8"
server_key: "9a0280e6555a5804075f61e7ff8bc394"
server_key_iv: "5393bf675e9f6ee5a86d465292144825"
server_key_version: "encryptionkey@0"
I20260926 23:23:13.890049 2104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:13.902585 2104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:13.906505 2104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:13.906776 2104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:13.928735 2104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42589
I20260926 23:23:13.928781 2180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42589 every 8 connection(s)
I20260926 23:23:13.930063 2104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:13.933749 2181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:13.937683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 2104
I20260926 23:23:13.937886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:13.938665 388 external_mini_cluster.cc:1428] Setting key b028aacc7f70722e2d754bcdd5a1e9be
I20260926 23:23:13.947170 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Bootstrap starting.
I20260926 23:23:13.950913 2181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:13.952219 2181 log.cc:824] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:23:13 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:13.955046 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: No bootstrap required, opened a new log
I20260926 23:23:13.961424 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:13.940608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.0.97.62:42589)
0926 23:23:13.941422 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:13.941433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:13.941472 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:23:13.943020 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:13.943042 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:13.943439 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:13.943756 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:13.945633 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:13.946854 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:13.948540 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:13.949260 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:13.949388 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:13.954194 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:13.954202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:13.954216 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:13.954249 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:13.957364 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:13.958082 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:13.958089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:13.958095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:13.958263 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:13.958629 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:13.958634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:13.958637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:13.959163 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:13.959473 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:13.959927 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 23:23:13.960365 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":321,"threads_started":1}
I20260926 23:23:13.962931 2181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3adafb1f74f388afc630d73a30170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260926 23:23:13.963326 2181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:13.963445 2181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d3adafb1f74f388afc630d73a30170, State: Initialized, Role: FOLLOWER
I20260926 23:23:13.964046 2181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3adafb1f74f388afc630d73a30170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260926 23:23:13.964334 2181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:13.964468 2181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:13.964596 2181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:13.966740 2181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3adafb1f74f388afc630d73a30170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260926 23:23:13.967167 2181 leader_election.cc:302] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d3adafb1f74f388afc630d73a30170; no voters:
I20260926 23:23:13.967720 2181 leader_election.cc:288] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:13.967782 2186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:13.969077 2186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [term 1 LEADER]: Becoming Leader. State: Replica: 64d3adafb1f74f388afc630d73a30170, State: Running, Role: LEADER
I20260926 23:23:13.969552 2186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3adafb1f74f388afc630d73a30170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260926 23:23:13.969928 2181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:13.972139 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d3adafb1f74f388afc630d73a30170. Latest consensus state: current_term: 1 leader_uuid: "64d3adafb1f74f388afc630d73a30170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3adafb1f74f388afc630d73a30170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } } }
I20260926 23:23:13.972471 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:13.972674 2187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d3adafb1f74f388afc630d73a30170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3adafb1f74f388afc630d73a30170" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } } }
I20260926 23:23:13.972954 2187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:13.975489 2193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:13.979732 2193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:13.988094 2193 catalog_manager.cc:1347] Generated new cluster ID: 46f7172bf462484da31ae5a14e748a0c
I20260926 23:23:13.988227 2193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:14.005954 2193 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:14.007504 2193 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:14.018255 2193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Generated new TSK 0
I20260926 23:23:14.019074 2193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:14.073235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42589
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:14.423631 2207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:14.423905 2207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:14.424077 2207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:14.431304 2207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:14.431404 2207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:14.431556 2207 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:23:14.431638 2207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:23:14.431777 2207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:14.441432 2207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:14.442747 2207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:14.444100 2207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:14.457252 2212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:14.457187 2213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:14.457305 2215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:14.458314 2207 server_base.cc:1034] running on GCE node
I20260926 23:23:14.458894 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:14.459790 2207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:14.461009 2207 hybrid_clock.cc:648] HybridClock initialized: now 1790464994460971 us; error 53 us; skew 500 ppm
I20260926 23:23:14.461313 2207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:14 dist-test-slave-q5g8 krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:14.464426 2207 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:14.465935 2207 webserver.cc:460] Webserver started at http://127.0.97.1:45855/ using document root <none> and password file <none>
I20260926 23:23:14.466544 2207 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:14.466657 2207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:14.466892 2207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:14.469691 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "a2db04d1d1424ce5854e926a813b6868"
format_stamp: "Formatted at 2026-09-26 23:23:14 on dist-test-slave-q5g8"
server_key: "ab136c460ed4268d5b9c8d013e48f592"
server_key_iv: "e6391978d1cbe1ff439591ba103fc586"
server_key_version: "encryptionkey@0"
I20260926 23:23:14.470499 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a2db04d1d1424ce5854e926a813b6868"
format_stamp: "Formatted at 2026-09-26 23:23:14 on dist-test-slave-q5g8"
server_key: "ab136c460ed4268d5b9c8d013e48f592"
server_key_iv: "e6391978d1cbe1ff439591ba103fc586"
server_key_version: "encryptionkey@0"
I20260926 23:23:14.475920 2207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 23:23:14.479364 2222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:14.480867 2207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:14.480994 2207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "a2db04d1d1424ce5854e926a813b6868"
format_stamp: "Formatted at 2026-09-26 23:23:14 on dist-test-slave-q5g8"
server_key: "ab136c460ed4268d5b9c8d013e48f592"
server_key_iv: "e6391978d1cbe1ff439591ba103fc586"
server_key_version: "encryptionkey@0"
I20260926 23:23:14.481159 2207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:14.504042 2207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:14.508358 2207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:14.508597 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:14.511363 2207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:14.511514 2207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:14.511657 2207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:14.511837 2207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 23:23:14 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:14.533404 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:14.511892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37357 (local address 127.0.97.62:42589)
0926 23:23:14.512326 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:14.512332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:14.513670 (+ 1338us) server_negotiation.cc:407] Connection header received
0926 23:23:14.515614 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:14.515620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:14.515711 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:14.515854 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:14.518305 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.519337 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.522621 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.523278 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.523414 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:14.527758 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:14.527763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:14.527766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:14.527796 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:14.530519 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:14.531218 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:14.531227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:14.531230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:14.531338 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:14.531733 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:14.531738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:14.531741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:14.531995 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:14.532144 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:14.532824 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 23:23:14.533026 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":236,"threads_started":1}
I20260926 23:23:14.534724 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:14.512533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42589 (local address 127.0.97.1:37357)
0926 23:23:14.513482 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:14.513529 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:14.515331 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:14.516149 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:14.516166 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:14.517005 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:14.518097 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:14.518138 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.519519 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.519527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:14.522422 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:14.522447 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.523513 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.523521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:14.523867 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:14.524773 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:14.524809 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:14.527533 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:14.530662 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:14.530671 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:14.530690 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:14.531090 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:14.531443 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:14.531448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:14.531452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:14.531616 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:14.532123 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:14.532131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:14.532439 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 23:23:14.532730 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 23:23:14.533280 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":457,"threads_started":1}
I20260926 23:23:14.556066 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:14.541396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55233 (local address 127.0.97.62:42589)
0926 23:23:14.541602 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:14.541608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:14.542653 (+ 1045us) server_negotiation.cc:407] Connection header received
0926 23:23:14.542843 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:14.542846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:14.542902 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:14.543011 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:14.543904 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.544578 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.546579 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.547361 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.547520 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:14.549852 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:14.549859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:14.549865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:14.549909 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:14.553310 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:14.553983 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:14.554000 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:14.554003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:14.554179 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:14.554681 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:14.554686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:14.554690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:14.554949 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:14.555111 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:14.555650 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 23:23:14.555796 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 23:23:14.556643 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:14.541685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42589 (local address 127.0.97.1:55233)
0926 23:23:14.542526 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:14.542551 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:14.542724 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:14.543057 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:14.543061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:14.543508 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:14.543772 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:14.543780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.544718 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.544726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:14.546348 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:14.546360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:14.547629 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:14.547637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:14.547806 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:14.548537 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:14.548555 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:14.549595 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:14.553443 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:14.553449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:14.553453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:14.553868 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:14.554313 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:14.554321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:14.554328 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:14.554549 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:14.555094 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:14.555102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:14.555365 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 23:23:14.555557 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 23:23:14.556004 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":479,"threads_started":1}
I20260926 23:23:14.568737 2207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39193
I20260926 23:23:14.568784 2342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39193 every 8 connection(s)
I20260926 23:23:14.570017 2207 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:14.572084 2343 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42589
I20260926 23:23:14.572268 2343 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:14.572914 2343 heartbeater.cc:511] Master 127.0.97.62:42589 requested a full tablet report, sending...
I20260926 23:23:14.573844 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 2207
I20260926 23:23:14.574036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:14.574822 388 external_mini_cluster.cc:1428] Setting key 8139466c24fe0ca771b6a72b1462dfb8
I20260926 23:23:14.575136 2136 ts_manager.cc:194] Registered new tserver with Master: a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193)
I20260926 23:23:14.576927 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37357
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:14.617121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42589
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:14.931334 2347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:14.931643 2347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:14.931823 2347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:14.939170 2347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:14.939308 2347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:14.939463 2347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:23:14.939553 2347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:23:14.939687 2347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:14.949198 2347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:14.950398 2347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:14.951689 2347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:14.964421 2352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:14.964532 2355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:14.964509 2353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:14.965734 2347 server_base.cc:1034] running on GCE node
I20260926 23:23:14.966358 2347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:14.967311 2347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:14.968545 2347 hybrid_clock.cc:648] HybridClock initialized: now 1790464994968504 us; error 54 us; skew 500 ppm
I20260926 23:23:14.968905 2347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:14 dist-test-slave-q5g8 krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:14.972155 2347 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:14.973575 2347 webserver.cc:460] Webserver started at http://127.0.97.2:44071/ using document root <none> and password file <none>
I20260926 23:23:14.974198 2347 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:14.974296 2347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:14.974548 2347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:14.977391 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3c2c4e194a34a40829ae725df3db6c2"
format_stamp: "Formatted at 2026-09-26 23:23:14 on dist-test-slave-q5g8"
server_key: "0a06f62340ee8c566d528b35eebfa024"
server_key_iv: "929df9134541ac3d6239be038ed5e785"
server_key_version: "encryptionkey@0"
I20260926 23:23:14.978097 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c2c4e194a34a40829ae725df3db6c2"
format_stamp: "Formatted at 2026-09-26 23:23:14 on dist-test-slave-q5g8"
server_key: "0a06f62340ee8c566d528b35eebfa024"
server_key_iv: "929df9134541ac3d6239be038ed5e785"
server_key_version: "encryptionkey@0"
I20260926 23:23:14.983120 2347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:14.986460 2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:14.987802 2347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:23:14.987965 2347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "f3c2c4e194a34a40829ae725df3db6c2"
format_stamp: "Formatted at 2026-09-26 23:23:14 on dist-test-slave-q5g8"
server_key: "0a06f62340ee8c566d528b35eebfa024"
server_key_iv: "929df9134541ac3d6239be038ed5e785"
server_key_version: "encryptionkey@0"
I20260926 23:23:14.988178 2347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:15.008939 2347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:15.013818 2347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:15.014151 2347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:15.016993 2347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:15.017145 2347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:15.017324 2347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:15.017488 2347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:23:15 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:15.040073 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.017336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48189 (local address 127.0.97.62:42589)
0926 23:23:15.017531 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.017538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.018912 (+ 1374us) server_negotiation.cc:407] Connection header received
0926 23:23:15.020932 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.020939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.021036 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.021194 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:15.023116 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.024455 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.026980 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.027567 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.027704 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.032318 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:15.032329 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:15.032338 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:15.032401 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:15.035905 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.036841 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.036849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.036855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.037018 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.037796 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.037805 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.037812 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.038280 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:15.038455 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.039149 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.039464 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:23:15.040539 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.017882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42589 (local address 127.0.97.2:48189)
0926 23:23:15.018725 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.018773 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.020594 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.021435 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.021442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.021936 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:15.022731 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.022749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.024712 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.024719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.026761 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.026781 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.027835 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.027846 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.028068 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.028806 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:15.028827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:15.032001 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:15.036054 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.036064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.036087 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.036673 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.037197 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.037207 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.037214 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.037588 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.038409 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:15.038415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:15.038737 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.038960 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.039333 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":378,"threads_started":1}
I20260926 23:23:15.064524 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.046964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36073 (local address 127.0.97.62:42589)
0926 23:23:15.047217 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.047231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.047863 (+ 632us) server_negotiation.cc:407] Connection header received
0926 23:23:15.048109 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.048117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.048206 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.048316 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:15.049745 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.050473 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.053269 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.054160 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.054339 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.056925 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:15.056936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:15.056945 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:15.057000 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:15.061297 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.061973 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.061982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.061989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.062183 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.062578 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.062586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.062591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.063172 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:15.063376 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.063781 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.064044 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:23:15.064713 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.046890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42589 (local address 127.0.97.2:36073)
0926 23:23:15.047724 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.047752 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.047943 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.048342 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.048352 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.048932 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:15.049557 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.049578 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.050620 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.050629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.053033 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.053055 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.054390 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.054399 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.054616 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.055370 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:15.055403 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:15.056600 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:15.061456 (+ 4856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.061459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.061462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.061825 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.062301 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.062304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.062306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.062415 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.063309 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:15.063312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:15.063491 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.063596 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.064033 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":394,"threads_started":1}
I20260926 23:23:15.071314 2347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42247
I20260926 23:23:15.071349 2481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42247 every 8 connection(s)
I20260926 23:23:15.072635 2347 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:15.074254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 2347
I20260926 23:23:15.074436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:15.074821 2482 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42589
I20260926 23:23:15.075049 2482 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:15.075079 388 external_mini_cluster.cc:1428] Setting key 202cdc096ac4a67c4778a11fc4958a0e
I20260926 23:23:15.075917 2482 heartbeater.cc:511] Master 127.0.97.62:42589 requested a full tablet report, sending...
I20260926 23:23:15.077438 2136 ts_manager.cc:194] Registered new tserver with Master: f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247)
I20260926 23:23:15.078739 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48189
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:15.120328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42589
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:15.435964 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:15.436233 2486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:15.436406 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:15.444021 2486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:15.444144 2486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:15.444324 2486 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:23:15.444412 2486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:23:15.444550 2486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:15.454372 2486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:15.455680 2486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:15.457046 2486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:15.470036 2491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:15.469951 2494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:15.470103 2492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:15.471275 2486 server_base.cc:1034] running on GCE node
I20260926 23:23:15.471850 2486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:15.472770 2486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:15.473971 2486 hybrid_clock.cc:648] HybridClock initialized: now 1790464995473931 us; error 36 us; skew 500 ppm
I20260926 23:23:15.474248 2486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:15 dist-test-slave-q5g8 krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:15.477450 2486 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:15.479002 2486 webserver.cc:460] Webserver started at http://127.0.97.3:41169/ using document root <none> and password file <none>
I20260926 23:23:15.479637 2486 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:15.479733 2486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:15.479971 2486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:15.482779 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "3311dc5268f44c8799ad3854f0abc741"
format_stamp: "Formatted at 2026-09-26 23:23:15 on dist-test-slave-q5g8"
server_key: "c6ec33ee17656a6f8f925db274aed1b6"
server_key_iv: "a47e9970152a08011f5d4c6ae1b6cb9a"
server_key_version: "encryptionkey@0"
I20260926 23:23:15.483516 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3311dc5268f44c8799ad3854f0abc741"
format_stamp: "Formatted at 2026-09-26 23:23:15 on dist-test-slave-q5g8"
server_key: "c6ec33ee17656a6f8f925db274aed1b6"
server_key_iv: "a47e9970152a08011f5d4c6ae1b6cb9a"
server_key_version: "encryptionkey@0"
I20260926 23:23:15.489120 2486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:23:15.492419 2501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:15.493893 2486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:15.494071 2486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "3311dc5268f44c8799ad3854f0abc741"
format_stamp: "Formatted at 2026-09-26 23:23:15 on dist-test-slave-q5g8"
server_key: "c6ec33ee17656a6f8f925db274aed1b6"
server_key_iv: "a47e9970152a08011f5d4c6ae1b6cb9a"
server_key_version: "encryptionkey@0"
I20260926 23:23:15.494282 2486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:15.515064 2486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:15.518955 2486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:15.519193 2486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:15.521917 2486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:15.522017 2486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:15.522174 2486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:15.522293 2486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:23:15 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:15.549835 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.523089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42589 (local address 127.0.97.3:35029)
0926 23:23:15.524015 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.524077 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.526869 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.527811 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.527828 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.528677 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:15.529661 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.529693 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.531126 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.531134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.533122 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.533133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.534030 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.534040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.534465 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.535428 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:15.535474 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:15.539494 (+ 4020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:15.544224 (+ 4730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.544236 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.544262 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.545031 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.545709 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.545714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.545718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.546026 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.546834 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:15.546840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:15.547209 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.547590 (+ 381us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.548198 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":373,"threads_started":1}
I20260926 23:23:15.551028 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.522583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35029 (local address 127.0.97.62:42589)
0926 23:23:15.522783 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.522790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.524225 (+ 1435us) server_negotiation.cc:407] Connection header received
0926 23:23:15.527237 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.527244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.527352 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.527481 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:15.529860 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.530967 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.533268 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.533745 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.533884 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.539794 (+ 5910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:15.539803 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:15.539811 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:15.539852 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:15.544001 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.545330 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.545335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.545338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.545553 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.546187 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.546196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.546202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.546687 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:15.546887 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.550425 (+ 3538us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.550621 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:23:15.562409 2486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40721
I20260926 23:23:15.562456 2620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40721 every 8 connection(s)
I20260926 23:23:15.563768 2486 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:15.566102 2621 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42589
I20260926 23:23:15.566336 2621 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:15.567025 2621 heartbeater.cc:511] Master 127.0.97.62:42589 requested a full tablet report, sending...
I20260926 23:23:15.568110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 2486
I20260926 23:23:15.568290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:15.568948 388 external_mini_cluster.cc:1428] Setting key ecc619c43d4f4045a5b877985e84fb9c
I20260926 23:23:15.569932 2136 ts_manager.cc:194] Registered new tserver with Master: 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721)
I20260926 23:23:15.570874 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35029
I20260926 23:23:15.572017 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.557198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42589 (local address 127.0.97.3:47063)
0926 23:23:15.557979 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.558002 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.558226 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.558748 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.558751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.559487 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:15.559743 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.559752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.561168 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.561173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.562437 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.562451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.563259 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.563266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.563419 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.564040 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:15.564059 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:15.565132 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:15.569090 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.569098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.569104 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.569602 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.570039 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.570044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.570048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.570182 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.570887 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:15.570891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:15.571066 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.571206 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.571549 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":431,"threads_started":1}
I20260926 23:23:15.572932 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.557266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47063 (local address 127.0.97.62:42589)
0926 23:23:15.557502 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.557508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.558098 (+ 590us) server_negotiation.cc:407] Connection header received
0926 23:23:15.558394 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.558401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.558501 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.558609 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:15.559891 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.561016 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.562578 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.563004 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.563168 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.565393 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:15.565400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:15.565404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:15.565439 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:15.568945 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.569755 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.569760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.569764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.569921 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.570306 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.570311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.570314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.570693 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:15.570830 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.572307 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.572546 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":84}
I20260926 23:23:15.574260 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:23:15.580360 2343 heartbeater.cc:503] Master 127.0.97.62:42589 was elected leader, sending a full tablet report...
Sep 26 23:23:15 dist-test-slave-q5g8 krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:23:15 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:15.622104 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.608871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.0.97.62:42589)
0926 23:23:15.609092 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.609100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.609326 (+ 226us) server_negotiation.cc:407] Connection header received
0926 23:23:15.609562 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.609573 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.609640 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.609740 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:15.611122 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.611987 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.613287 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.613697 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.613829 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.617503 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:15.617507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:15.617511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:15.617540 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:15.619849 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.620421 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.620425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.620427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.620559 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.620980 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.620984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.620986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.621247 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:15.621478 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.621644 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.621798 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:23:15.626129 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:15.632987 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:15.656138 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.644945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40721 (local address 127.0.0.1:50374)
0926 23:23:15.646335 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.646384 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.646597 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.649418 (+ 2821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.649421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.649437 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:15.650094 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.650111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.651766 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.651771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.653899 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.653913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.655175 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.655181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.655355 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.655418 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.655548 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.655724 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"mutex_wait_us":4,"thread_start_us":2834,"threads_started":1}
I20260926 23:23:15.657157 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.645192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.0.97.3:40721)
0926 23:23:15.648867 (+ 3675us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.648877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.648912 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:23:15.649005 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.649014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.649231 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.650039 (+ 808us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:15.650350 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.651604 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.654071 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.654977 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.655145 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.655386 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.656644 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.656845 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2266,"threads_started":1}
I20260926 23:23:15.657279 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.647902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42247 (local address 127.0.0.1:36102)
0926 23:23:15.648890 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.648908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.649031 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.649570 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.649575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.649599 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:15.650319 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.650332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.652171 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.652178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.654640 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.654652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.656455 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.656461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.656655 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.656686 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.656794 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.656925 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":109,"thread_start_us":293,"threads_started":1}
I20260926 23:23:15.657703 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.645799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.0.97.1:39193)
0926 23:23:15.647251 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.647264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.648786 (+ 1522us) server_negotiation.cc:407] Connection header received
0926 23:23:15.648894 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.648899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.649178 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.649368 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:15.650590 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.652093 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.655083 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.656729 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.656843 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.657041 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.657271 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.657439 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":1098,"threads_started":1}
I20260926 23:23:15.657711 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.645163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:54402)
0926 23:23:15.648484 (+ 3321us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.648503 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.648624 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.649394 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.649404 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.649432 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:15.650344 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.650359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.652274 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.652283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.654925 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.654936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.656908 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.656915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.657083 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.657112 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.657218 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.657349 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3175,"mutex_wait_us":28,"thread_start_us":834,"threads_started":1}
I20260926 23:23:15.657972 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.648132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.0.97.2:42247)
0926 23:23:15.648786 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.648794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.649002 (+ 208us) server_negotiation.cc:407] Connection header received
0926 23:23:15.649169 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.649175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.649393 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.649571 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:15.650529 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.652009 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.654809 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.656160 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.656289 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.656467 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.657527 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.657675 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":344,"threads_started":1}
I20260926 23:23:15.661146 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 00fa9d7573ea41aba9a78fc23561d704 (DEFAULT_TABLE table=test-table [id=dbc800539713403eb920595349e2bd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:15.661147 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 00fa9d7573ea41aba9a78fc23561d704 (DEFAULT_TABLE table=test-table [id=dbc800539713403eb920595349e2bd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:15.661109 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 00fa9d7573ea41aba9a78fc23561d704 (DEFAULT_TABLE table=test-table [id=dbc800539713403eb920595349e2bd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:15.662381 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fa9d7573ea41aba9a78fc23561d704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:15.662467 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fa9d7573ea41aba9a78fc23561d704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:15.662473 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fa9d7573ea41aba9a78fc23561d704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:15.673126 2640 tablet_bootstrap.cc:492] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Bootstrap starting.
I20260926 23:23:15.676122 2641 tablet_bootstrap.cc:492] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Bootstrap starting.
I20260926 23:23:15.677716 2640 tablet_bootstrap.cc:654] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:15.678824 2641 tablet_bootstrap.cc:654] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:15.679322 2640 log.cc:824] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:15.679883 2641 log.cc:824] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:15.682040 2640 tablet_bootstrap.cc:492] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: No bootstrap required, opened a new log
I20260926 23:23:15.682247 2642 tablet_bootstrap.cc:492] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Bootstrap starting.
I20260926 23:23:15.682340 2640 ts_tablet_manager.cc:1397] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260926 23:23:15.685214 2641 tablet_bootstrap.cc:492] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: No bootstrap required, opened a new log
I20260926 23:23:15.685467 2641 ts_tablet_manager.cc:1397] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 23:23:15.685796 2642 tablet_bootstrap.cc:654] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:15.686964 2642 log.cc:824] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:15.689864 2642 tablet_bootstrap.cc:492] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: No bootstrap required, opened a new log
I20260926 23:23:15.690116 2642 ts_tablet_manager.cc:1397] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 23:23:15.692855 2640 raft_consensus.cc:348] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.693342 2640 raft_consensus.cc:374] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:15.693444 2640 raft_consensus.cc:729] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2c4e194a34a40829ae725df3db6c2, State: Initialized, Role: FOLLOWER
I20260926 23:23:15.694059 2640 consensus_queue.cc:260] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.694036 2641 raft_consensus.cc:348] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.694515 2641 raft_consensus.cc:374] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:15.694648 2641 raft_consensus.cc:729] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Initialized, Role: FOLLOWER
I20260926 23:23:15.695197 2641 consensus_queue.cc:260] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.696856 2641 ts_tablet_manager.cc:1428] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 23:23:15.697722 2482 heartbeater.cc:503] Master 127.0.97.62:42589 was elected leader, sending a full tablet report...
I20260926 23:23:15.698376 2640 ts_tablet_manager.cc:1428] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260926 23:23:15.700455 2642 raft_consensus.cc:348] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.700904 2642 raft_consensus.cc:374] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:15.701021 2642 raft_consensus.cc:729] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3311dc5268f44c8799ad3854f0abc741, State: Initialized, Role: FOLLOWER
I20260926 23:23:15.701598 2642 consensus_queue.cc:260] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.702603 2621 heartbeater.cc:503] Master 127.0.97.62:42589 was elected leader, sending a full tablet report...
I20260926 23:23:15.703153 2642 ts_tablet_manager.cc:1428] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 23:23:15.762108 2647 raft_consensus.cc:482] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:15.762373 2647 raft_consensus.cc:504] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.763942 2647 leader_election.cc:288] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721), f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247)
I20260926 23:23:15.773226 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.764348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34805 (local address 127.0.97.3:40721)
0926 23:23:15.764546 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.764554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.765313 (+ 759us) server_negotiation.cc:407] Connection header received
0926 23:23:15.765926 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.765953 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.766028 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.766120 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:15.766683 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.767805 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.770729 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.771982 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.772119 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.772246 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.772826 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.772967 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:23:15.773236 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.764348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48533 (local address 127.0.97.2:42247)
0926 23:23:15.764546 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.764553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.765367 (+ 814us) server_negotiation.cc:407] Connection header received
0926 23:23:15.765540 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.765547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.765625 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.765727 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:15.766849 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.767902 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.770729 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.771996 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.772118 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.772246 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.772830 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.772967 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:23:15.773445 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.764239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40721 (local address 127.0.97.1:34805)
0926 23:23:15.765181 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.765197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.765378 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.766161 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.766166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.766189 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:15.766518 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.766529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.767947 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.767968 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.770561 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.770576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.772257 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.772264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.772554 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.772645 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.772761 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.772949 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":208,"spinlock_wait_cycles":14336,"thread_start_us":534,"threads_started":1}
I20260926 23:23:15.773469 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.764238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42247 (local address 127.0.97.1:48533)
0926 23:23:15.765176 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.765197 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.765378 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.765774 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.765780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.765807 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:15.766163 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.766174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.768014 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.768024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.770566 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.770576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.772275 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.772280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.772561 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.772643 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.772761 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.772949 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":290,"spinlock_wait_cycles":23168,"thread_start_us":489,"threads_started":1}
I20260926 23:23:15.774674 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fa9d7573ea41aba9a78fc23561d704" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741" is_pre_election: true
I20260926 23:23:15.774676 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fa9d7573ea41aba9a78fc23561d704" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2" is_pre_election: true
I20260926 23:23:15.775255 2437 raft_consensus.cc:2454] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 0.
I20260926 23:23:15.775251 2571 raft_consensus.cc:2454] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 0.
I20260926 23:23:15.775976 2226 leader_election.cc:302] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3311dc5268f44c8799ad3854f0abc741, a2db04d1d1424ce5854e926a813b6868, f3c2c4e194a34a40829ae725df3db6c2; no voters:
I20260926 23:23:15.776513 2647 raft_consensus.cc:2781] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:15.776716 2647 raft_consensus.cc:482] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:15.776852 2647 raft_consensus.cc:3037] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:15.779188 2647 raft_consensus.cc:504] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.779808 2647 leader_election.cc:288] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 election: Requested vote from peers 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721), f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247)
I20260926 23:23:15.780313 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fa9d7573ea41aba9a78fc23561d704" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741"
I20260926 23:23:15.780313 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fa9d7573ea41aba9a78fc23561d704" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2"
I20260926 23:23:15.780527 2571 raft_consensus.cc:3037] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:15.780560 2437 raft_consensus.cc:3037] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:15.783902 2571 raft_consensus.cc:2454] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 1.
I20260926 23:23:15.783902 2437 raft_consensus.cc:2454] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 1.
I20260926 23:23:15.784315 2226 leader_election.cc:302] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3311dc5268f44c8799ad3854f0abc741, a2db04d1d1424ce5854e926a813b6868, f3c2c4e194a34a40829ae725df3db6c2; no voters:
I20260926 23:23:15.784673 2647 raft_consensus.cc:2781] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:15.785228 2647 raft_consensus.cc:686] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 1 LEADER]: Becoming Leader. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Running, Role: LEADER
I20260926 23:23:15.785844 2647 consensus_queue.cc:237] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } }
I20260926 23:23:15.790462 2135 catalog_manager.cc:5495] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2db04d1d1424ce5854e926a813b6868 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } health_report { overall_health: UNKNOWN } } }
W20260926 23:23:15.815548 2622 tablet.cc:2369] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:23:15.822363 2344 tablet.cc:2369] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:23:15.824847 2483 tablet.cc:2369] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 23:23:15 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:15.836016 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.823036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.0.97.62:42589)
0926 23:23:15.823200 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:15.823204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:15.823843 (+ 639us) server_negotiation.cc:407] Connection header received
0926 23:23:15.824054 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:15.824058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:15.824109 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:15.824203 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:15.825431 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.826183 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.827433 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.827825 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.827991 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.831076 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:15.831081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:15.831084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:15.831109 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:15.833520 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.834190 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.834197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.834203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.834309 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:15.834658 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:15.834666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:15.834669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:15.834895 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:15.835032 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:15.835419 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 23:23:15.835710 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":64}
I20260926 23:23:15.836059 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:15.823171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42589 (local address 127.0.0.1:47194)
0926 23:23:15.823739 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:15.823755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:15.823923 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:15.824221 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:15.824225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:15.825014 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:15.825289 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.825299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.826312 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.826320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.827288 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:15.827297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:15.828028 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:15.828034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:15.828197 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:15.828835 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:15.828851 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:15.830908 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:15.833642 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.833655 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.833662 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.834074 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.834410 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:15.834417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:15.834420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:15.834560 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:15.835028 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:15.835035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:15.835181 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 23:23:15.835346 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 23:23:15.835612 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":318,"threads_started":1}
I20260926 23:23:15.840762 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47194:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:23:15.841732 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:15.848140 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 69104561b56b4ab7a14babf70356d258 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12861a1353dd4429be17383babc58d79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:23:15.848402 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 69104561b56b4ab7a14babf70356d258 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12861a1353dd4429be17383babc58d79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:23:15.848623 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 69104561b56b4ab7a14babf70356d258 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12861a1353dd4429be17383babc58d79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:23:15.849218 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69104561b56b4ab7a14babf70356d258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:15.849288 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69104561b56b4ab7a14babf70356d258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:15.849431 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69104561b56b4ab7a14babf70356d258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:15.855110 2640 tablet_bootstrap.cc:492] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: Bootstrap starting.
I20260926 23:23:15.856374 2642 tablet_bootstrap.cc:492] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: Bootstrap starting.
I20260926 23:23:15.856551 2640 tablet_bootstrap.cc:654] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:15.858186 2642 tablet_bootstrap.cc:654] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:15.859467 2641 tablet_bootstrap.cc:492] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: Bootstrap starting.
I20260926 23:23:15.861534 2641 tablet_bootstrap.cc:654] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:15.862224 2642 tablet_bootstrap.cc:492] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: No bootstrap required, opened a new log
I20260926 23:23:15.862403 2642 ts_tablet_manager.cc:1397] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 23:23:15.862586 2640 tablet_bootstrap.cc:492] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: No bootstrap required, opened a new log
I20260926 23:23:15.862730 2640 ts_tablet_manager.cc:1397] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 23:23:15.863274 2642 raft_consensus.cc:348] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:15.863502 2642 raft_consensus.cc:374] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:15.863620 2642 raft_consensus.cc:729] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3311dc5268f44c8799ad3854f0abc741, State: Initialized, Role: FOLLOWER
I20260926 23:23:15.863523 2640 raft_consensus.cc:348] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:15.863730 2641 tablet_bootstrap.cc:492] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: No bootstrap required, opened a new log
I20260926 23:23:15.863866 2641 ts_tablet_manager.cc:1397] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 23:23:15.863853 2640 raft_consensus.cc:374] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:15.863955 2640 raft_consensus.cc:729] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2c4e194a34a40829ae725df3db6c2, State: Initialized, Role: FOLLOWER
I20260926 23:23:15.863873 2642 consensus_queue.cc:260] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:15.864531 2642 ts_tablet_manager.cc:1428] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 23:23:15.864351 2648 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:15.864758 2648 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:15.864632 2640 consensus_queue.cc:260] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:15.865267 2640 ts_tablet_manager.cc:1428] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:15.865150 2646 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:15.865535 2646 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:15.866746 2641 raft_consensus.cc:348] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:15.866990 2641 raft_consensus.cc:374] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:15.867095 2641 raft_consensus.cc:729] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Initialized, Role: FOLLOWER
I20260926 23:23:15.867329 2641 consensus_queue.cc:260] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:15.867786 2647 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:15.868072 2647 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:15.868609 2641 ts_tablet_manager.cc:1428] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 23:23:16.097196 2647 raft_consensus.cc:482] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:16.097532 2647 raft_consensus.cc:504] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.098340 2647 leader_election.cc:288] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247), 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721)
I20260926 23:23:16.098875 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69104561b56b4ab7a14babf70356d258" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2" is_pre_election: true
I20260926 23:23:16.098927 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69104561b56b4ab7a14babf70356d258" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741" is_pre_election: true
I20260926 23:23:16.099177 2571 raft_consensus.cc:2454] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 0.
I20260926 23:23:16.099217 2437 raft_consensus.cc:2454] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 0.
I20260926 23:23:16.099661 2226 leader_election.cc:302] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3311dc5268f44c8799ad3854f0abc741, a2db04d1d1424ce5854e926a813b6868; no voters:
I20260926 23:23:16.099922 2647 raft_consensus.cc:2781] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:16.100024 2647 raft_consensus.cc:482] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:16.100153 2647 raft_consensus.cc:3037] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:16.102427 2647 raft_consensus.cc:504] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.102984 2647 leader_election.cc:288] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247), 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721)
I20260926 23:23:16.103484 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69104561b56b4ab7a14babf70356d258" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2"
I20260926 23:23:16.103499 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69104561b56b4ab7a14babf70356d258" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741"
I20260926 23:23:16.103682 2437 raft_consensus.cc:3037] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:16.103793 2571 raft_consensus.cc:3037] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:16.105844 2437 raft_consensus.cc:2454] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 1.
I20260926 23:23:16.106081 2571 raft_consensus.cc:2454] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 1.
I20260926 23:23:16.106317 2223 leader_election.cc:302] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2db04d1d1424ce5854e926a813b6868, f3c2c4e194a34a40829ae725df3db6c2; no voters:
I20260926 23:23:16.106649 2647 raft_consensus.cc:2781] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:16.106850 2647 raft_consensus.cc:686] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [term 1 LEADER]: Becoming Leader. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Running, Role: LEADER
I20260926 23:23:16.107100 2647 consensus_queue.cc:237] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.108793 2651 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: TxnStatusTablet state changed. Reason: New leader a2db04d1d1424ce5854e926a813b6868. Latest consensus state: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:16.109189 2651 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:23:16.109889 2663 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:23:16.110157 2663 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:23:16.110121 2134 catalog_manager.cc:5495] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2db04d1d1424ce5854e926a813b6868 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260926 23:23:16.156258 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.150040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.97.1:39193)
0926 23:23:16.150239 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.150243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.150526 (+ 283us) server_negotiation.cc:407] Connection header received
0926 23:23:16.150797 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.150801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.150890 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.151028 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:16.151413 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.152233 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.154146 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.154550 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.154644 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.155217 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.155727 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.155833 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.155979 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.156041 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:23:16.156409 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.150096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:54416)
0926 23:23:16.150434 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:16.150455 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:16.150613 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:16.151076 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:16.151081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:16.151094 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:16.151285 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.151295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.152355 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.152362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.154003 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.154021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.154705 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.154713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.154941 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.155023 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.155835 (+ 812us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.155854 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 23:23:16.155953 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 23:23:16.156101 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 23:23:16.163884 2437 raft_consensus.cc:1261] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Refusing update from remote peer a2db04d1d1424ce5854e926a813b6868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:16.164091 2571 raft_consensus.cc:1261] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Refusing update from remote peer a2db04d1d1424ce5854e926a813b6868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:16.164951 2647 consensus_queue.cc:1035] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:16.165263 2651 consensus_queue.cc:1035] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:16.173141 2646 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: TxnStatusTablet state changed. Reason: New leader a2db04d1d1424ce5854e926a813b6868. Latest consensus state: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:16.173439 2646 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:16.176151 2666 mvcc.cc:205] Tried to move back new op lower bound from 7333744624278040576 to 7333744624057237504. Current Snapshot: MvccSnapshot[applied={T|T < 7333744624278040576}]
I20260926 23:23:16.177040 2651 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:16.177292 2651 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:23:16.178994 2646 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:16.179271 2646 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P f3c2c4e194a34a40829ae725df3db6c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:16.180320 2648 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: TxnStatusTablet state changed. Reason: New leader a2db04d1d1424ce5854e926a813b6868. Latest consensus state: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:16.180562 2648 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:16.185313 2647 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:16.185614 2647 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P a2db04d1d1424ce5854e926a813b6868: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:23:16.188321 2647 consensus_queue.cc:1035] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:16.200644 2648 tablet_replica.cc:439] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } } }
I20260926 23:23:16.201862 2648 tablet_replica.cc:442] T 69104561b56b4ab7a14babf70356d258 P 3311dc5268f44c8799ad3854f0abc741: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:23:16.212194 2647 consensus_queue.cc:1035] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:16.216655 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.208068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.97.1:39193)
0926 23:23:16.208257 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.208281 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.208466 (+ 185us) server_negotiation.cc:407] Connection header received
0926 23:23:16.208709 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.208716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.208806 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.208870 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:16.209609 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.210918 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.214303 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.214984 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.215085 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.215827 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.216005 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.216100 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.216257 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.216331 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:23:16.232967 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.227431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57753 (local address 127.0.97.1:39193)
0926 23:23:16.227579 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.227583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.228236 (+ 653us) server_negotiation.cc:407] Connection header received
0926 23:23:16.228434 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.228450 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.228505 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.228594 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:16.229102 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.229840 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.231246 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.231682 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.231812 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.232235 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.232345 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.232435 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.232564 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.232622 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":44}
I20260926 23:23:16.233183 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.227544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.97.1:57753)
0926 23:23:16.228142 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:16.228157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:16.228317 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:16.228613 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:16.228617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:16.228627 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:16.228967 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.228976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.229982 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.229992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.231111 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.231121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.231874 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.231880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.232004 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.232049 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.232435 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.232444 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:23:16.232538 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 23:23:16.232656 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":4,"thread_start_us":323,"threads_started":1}
I20260926 23:23:16.297092 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:16.297394 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:16.300988 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Sending DeleteTablet for 3 replicas of tablet 00fa9d7573ea41aba9a78fc23561d704
I20260926 23:23:16.302026 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fa9d7573ea41aba9a78fc23561d704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50374
I20260926 23:23:16.302453 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fa9d7573ea41aba9a78fc23561d704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54402
I20260926 23:23:16.303346 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 23:23:16.303485 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fa9d7573ea41aba9a78fc23561d704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36102
W20260926 23:23:16.303984 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:16.309326 2685 tablet_replica.cc:330] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: stopping tablet replica
I20260926 23:23:16.309850 2685 raft_consensus.cc:2229] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:16.311529 2685 raft_consensus.cc:2258] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:16.312587 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa293625be64799ab89f747048d5b7a (DEFAULT_TABLE table=test-table [id=182ff3b114ef435dbef289af5fbebdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:16.313261 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa293625be64799ab89f747048d5b7a (DEFAULT_TABLE table=test-table [id=182ff3b114ef435dbef289af5fbebdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:16.313820 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa293625be64799ab89f747048d5b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:16.313727 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa293625be64799ab89f747048d5b7a (DEFAULT_TABLE table=test-table [id=182ff3b114ef435dbef289af5fbebdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:16.314306 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa293625be64799ab89f747048d5b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:16.315114 2685 ts_tablet_manager.cc:1905] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:16.322048 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa293625be64799ab89f747048d5b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:16.331393 2641 tablet_bootstrap.cc:492] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: Bootstrap starting.
I20260926 23:23:16.333196 2685 ts_tablet_manager.cc:1918] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:16.333271 2641 tablet_bootstrap.cc:654] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:16.333379 2685 log.cc:1196] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/00fa9d7573ea41aba9a78fc23561d704
I20260926 23:23:16.334036 2685 ts_tablet_manager.cc:1939] T 00fa9d7573ea41aba9a78fc23561d704 P a2db04d1d1424ce5854e926a813b6868: Deleting consensus metadata
I20260926 23:23:16.336131 2120 catalog_manager.cc:4841] TS a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193): tablet 00fa9d7573ea41aba9a78fc23561d704 (table test-table [id=dbc800539713403eb920595349e2bd9d]) successfully deleted
I20260926 23:23:16.341899 2642 tablet_bootstrap.cc:492] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: Bootstrap starting.
I20260926 23:23:16.342717 2640 tablet_bootstrap.cc:492] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: Bootstrap starting.
I20260926 23:23:16.344012 2642 tablet_bootstrap.cc:654] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:16.344763 2640 tablet_bootstrap.cc:654] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:16.345281 2686 tablet_replica.cc:330] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: stopping tablet replica
I20260926 23:23:16.345820 2686 raft_consensus.cc:2229] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:16.348577 2641 tablet_bootstrap.cc:492] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: No bootstrap required, opened a new log
I20260926 23:23:16.348810 2641 ts_tablet_manager.cc:1397] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260926 23:23:16.349823 2641 raft_consensus.cc:348] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.350173 2641 raft_consensus.cc:374] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:16.350276 2641 raft_consensus.cc:729] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Initialized, Role: FOLLOWER
I20260926 23:23:16.350629 2641 consensus_queue.cc:260] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.352641 2641 ts_tablet_manager.cc:1428] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 23:23:16.346282 2686 raft_consensus.cc:2258] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:16.353853 2684 tablet_replica.cc:330] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: stopping tablet replica
I20260926 23:23:16.354295 2684 raft_consensus.cc:2229] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:16.354676 2684 raft_consensus.cc:2258] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:16.356695 2684 ts_tablet_manager.cc:1905] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:16.357046 2642 tablet_bootstrap.cc:492] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: No bootstrap required, opened a new log
I20260926 23:23:16.357210 2642 ts_tablet_manager.cc:1397] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20260926 23:23:16.357628 2686 ts_tablet_manager.cc:1905] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:16.358067 2642 raft_consensus.cc:348] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.358333 2642 raft_consensus.cc:374] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:16.358417 2642 raft_consensus.cc:729] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3311dc5268f44c8799ad3854f0abc741, State: Initialized, Role: FOLLOWER
I20260926 23:23:16.358680 2642 consensus_queue.cc:260] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.359206 2642 ts_tablet_manager.cc:1428] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 23:23:16.365490 2684 ts_tablet_manager.cc:1918] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:16.365648 2684 log.cc:1196] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/00fa9d7573ea41aba9a78fc23561d704
I20260926 23:23:16.366122 2684 ts_tablet_manager.cc:1939] T 00fa9d7573ea41aba9a78fc23561d704 P 3311dc5268f44c8799ad3854f0abc741: Deleting consensus metadata
I20260926 23:23:16.366613 2686 ts_tablet_manager.cc:1918] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:16.366770 2686 log.cc:1196] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/00fa9d7573ea41aba9a78fc23561d704
I20260926 23:23:16.367262 2686 ts_tablet_manager.cc:1939] T 00fa9d7573ea41aba9a78fc23561d704 P f3c2c4e194a34a40829ae725df3db6c2: Deleting consensus metadata
I20260926 23:23:16.367965 2123 catalog_manager.cc:4841] TS 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721): tablet 00fa9d7573ea41aba9a78fc23561d704 (table test-table [id=dbc800539713403eb920595349e2bd9d]) successfully deleted
I20260926 23:23:16.368716 2120 catalog_manager.cc:4841] TS f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247): tablet 00fa9d7573ea41aba9a78fc23561d704 (table test-table [id=dbc800539713403eb920595349e2bd9d]) successfully deleted
I20260926 23:23:16.369845 2640 tablet_bootstrap.cc:492] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: No bootstrap required, opened a new log
I20260926 23:23:16.370007 2640 ts_tablet_manager.cc:1397] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.004s
I20260926 23:23:16.370728 2640 raft_consensus.cc:348] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.370977 2640 raft_consensus.cc:374] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:16.371050 2640 raft_consensus.cc:729] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2c4e194a34a40829ae725df3db6c2, State: Initialized, Role: FOLLOWER
I20260926 23:23:16.371308 2640 consensus_queue.cc:260] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.371872 2640 ts_tablet_manager.cc:1428] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 23:23:16.510509 2648 raft_consensus.cc:482] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:16.510787 2648 raft_consensus.cc:504] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.512821 2648 leader_election.cc:288] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193), f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247)
I20260926 23:23:16.523577 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.513105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41403 (local address 127.0.97.1:39193)
0926 23:23:16.513337 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.513343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.513643 (+ 300us) server_negotiation.cc:407] Connection header received
0926 23:23:16.513964 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.513973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.514086 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.514175 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:16.515059 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.516761 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.520091 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.521939 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.522091 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.522262 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.522949 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.523168 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:23:16.524631 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.513041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.97.3:41403)
0926 23:23:16.513552 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:16.513570 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:16.513719 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:16.514423 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:16.514429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:16.514466 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:16.514837 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.514851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.517009 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.517018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.519857 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.519878 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.522426 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.522435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.522698 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.522751 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 23:23:16.523947 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0926 23:23:16.524162 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":264,"threads_started":1}
I20260926 23:23:16.526646 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa293625be64799ab89f747048d5b7a" candidate_uuid: "3311dc5268f44c8799ad3854f0abc741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2db04d1d1424ce5854e926a813b6868" is_pre_election: true
I20260926 23:23:16.527128 2298 raft_consensus.cc:2454] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3311dc5268f44c8799ad3854f0abc741 in term 0.
I20260926 23:23:16.527846 2502 leader_election.cc:302] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3311dc5268f44c8799ad3854f0abc741, a2db04d1d1424ce5854e926a813b6868; no voters:
I20260926 23:23:16.528326 2648 raft_consensus.cc:2781] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:16.528462 2648 raft_consensus.cc:482] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:16.528561 2648 raft_consensus.cc:3037] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:16.531040 2648 raft_consensus.cc:504] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.531692 2648 leader_election.cc:288] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [CANDIDATE]: Term 1 election: Requested vote from peers a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193), f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247)
I20260926 23:23:16.532022 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.513727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42247 (local address 127.0.97.3:45139)
0926 23:23:16.524963 (+ 11236us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:16.524989 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:16.525137 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:16.526101 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:16.526105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:16.526132 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:16.526314 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.526331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.527713 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.527718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.529635 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.529649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.531265 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.531270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.531458 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.531493 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:23:16.531592 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 23:23:16.531739 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11086,"thread_start_us":11923,"threads_started":1}
I20260926 23:23:16.532330 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa293625be64799ab89f747048d5b7a" candidate_uuid: "3311dc5268f44c8799ad3854f0abc741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2db04d1d1424ce5854e926a813b6868"
I20260926 23:23:16.532418 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.513981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45139 (local address 127.0.97.2:42247)
0926 23:23:16.525717 (+ 11736us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.525723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.525747 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:23:16.525812 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.525817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.525896 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.526082 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:16.526487 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.527582 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.529802 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.531085 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.531330 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.531452 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.531627 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.531818 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11591,"thread_start_us":11418,"threads_started":1}
I20260926 23:23:16.532547 2298 raft_consensus.cc:3037] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:16.533334 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa293625be64799ab89f747048d5b7a" candidate_uuid: "3311dc5268f44c8799ad3854f0abc741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2" is_pre_election: true
I20260926 23:23:16.533586 2437 raft_consensus.cc:2454] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3311dc5268f44c8799ad3854f0abc741 in term 0.
I20260926 23:23:16.533962 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa293625be64799ab89f747048d5b7a" candidate_uuid: "3311dc5268f44c8799ad3854f0abc741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2"
I20260926 23:23:16.534211 2436 raft_consensus.cc:3037] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:16.535871 2298 raft_consensus.cc:2454] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3311dc5268f44c8799ad3854f0abc741 in term 1.
I20260926 23:23:16.536235 2502 leader_election.cc:302] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3311dc5268f44c8799ad3854f0abc741, a2db04d1d1424ce5854e926a813b6868; no voters:
I20260926 23:23:16.536516 2648 raft_consensus.cc:2781] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:16.537070 2648 raft_consensus.cc:686] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 1 LEADER]: Becoming Leader. State: Replica: 3311dc5268f44c8799ad3854f0abc741, State: Running, Role: LEADER
I20260926 23:23:16.537523 2436 raft_consensus.cc:2454] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3311dc5268f44c8799ad3854f0abc741 in term 1.
I20260926 23:23:16.537493 2648 consensus_queue.cc:237] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:16.541256 2134 catalog_manager.cc:5495] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3311dc5268f44c8799ad3854f0abc741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } health_report { overall_health: HEALTHY } } }
I20260926 23:23:16.585320 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.575919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.0.97.3:40721)
0926 23:23:16.577685 (+ 1766us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.577691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.577713 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:23:16.577803 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.577808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.577902 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.578497 (+ 595us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:16.578826 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.580044 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.581996 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.582642 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.583220 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.583377 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.584208 (+ 831us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.584747 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.584799 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.584878 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1645,"thread_start_us":1489,"threads_started":1}
I20260926 23:23:16.617811 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.611643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.97.3:42619)
0926 23:23:16.612416 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:16.612433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:16.612555 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:16.612934 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:16.612938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:16.612947 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:16.613206 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.613213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.614674 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.614677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.615706 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.615717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.616750 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.616755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.616874 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.616940 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.617374 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.617382 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:23:16.617448 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 23:23:16.617545 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":405,"threads_started":1}
I20260926 23:23:16.617889 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.611605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42619 (local address 127.0.97.1:39193)
0926 23:23:16.611800 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.611808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.612546 (+ 738us) server_negotiation.cc:407] Connection header received
0926 23:23:16.612668 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.612673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.612789 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.612910 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:16.613358 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.614543 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.615847 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.616538 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.616708 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.617105 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.617263 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.617370 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.617491 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.617571 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:23:16.626549 2436 raft_consensus.cc:1261] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Refusing update from remote peer 3311dc5268f44c8799ad3854f0abc741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:16.626711 2298 raft_consensus.cc:1261] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Refusing update from remote peer 3311dc5268f44c8799ad3854f0abc741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:16.627297 2693 consensus_queue.cc:1035] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:16.627826 2648 consensus_queue.cc:1035] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:16.632717 2668 mvcc.cc:205] Tried to move back new op lower bound from 7333744626176184320 to 7333744625822134272. Current Snapshot: MvccSnapshot[applied={T|T < 7333744626176184320}]
I20260926 23:23:16.632952 2667 mvcc.cc:205] Tried to move back new op lower bound from 7333744626176184320 to 7333744625822134272. Current Snapshot: MvccSnapshot[applied={T|T < 7333744626176184320}]
I20260926 23:23:16.657068 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.650261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40721 (local address 127.0.97.1:37717)
0926 23:23:16.650658 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:16.650678 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:16.650790 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:16.651152 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:16.651158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:16.651170 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:16.651368 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.651384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.652773 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.652778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.654451 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:16.654466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.655816 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.655822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:16.656012 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.656062 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.656590 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.656606 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 23:23:16.656661 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 23:23:16.656835 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260926 23:23:16.657692 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:16.650541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37717 (local address 127.0.97.3:40721)
0926 23:23:16.650859 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:16.650864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:16.650882 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:23:16.650938 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:16.650944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:16.651020 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:16.651116 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:16.651544 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.652587 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.654646 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:16.655256 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:16.656223 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:16.656348 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:16.656469 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:16.657403 (+ 934us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:16.657440 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 23:23:16.657501 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 23:23:16.658702 2668 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 23:23:18.260784 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:18.261018 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:18.264935 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Sending DeleteTablet for 3 replicas of tablet 9fa293625be64799ab89f747048d5b7a
I20260926 23:23:18.265712 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa293625be64799ab89f747048d5b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54402
I20260926 23:23:18.265834 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa293625be64799ab89f747048d5b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260926 23:23:18.265902 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa293625be64799ab89f747048d5b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50374
I20260926 23:23:18.266458 2724 tablet_replica.cc:330] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: stopping tablet replica
I20260926 23:23:18.266770 2724 raft_consensus.cc:2229] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:18.266986 2725 tablet_replica.cc:330] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: stopping tablet replica
I20260926 23:23:18.267041 2724 raft_consensus.cc:2258] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:18.267159 2726 tablet_replica.cc:330] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: stopping tablet replica
I20260926 23:23:18.267230 2725 raft_consensus.cc:2229] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:18.267375 2726 raft_consensus.cc:2229] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:18.267503 2725 raft_consensus.cc:2258] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:18.267982 2726 raft_consensus.cc:2258] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:18.268613 2725 ts_tablet_manager.cc:1905] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:18.269033 2726 ts_tablet_manager.cc:1905] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:18.273309 2724 ts_tablet_manager.cc:1905] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:18.273802 2725 ts_tablet_manager.cc:1918] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:23:18.273973 2725 log.cc:1196] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/9fa293625be64799ab89f747048d5b7a
I20260926 23:23:18.274508 2725 ts_tablet_manager.cc:1939] T 9fa293625be64799ab89f747048d5b7a P f3c2c4e194a34a40829ae725df3db6c2: Deleting consensus metadata
I20260926 23:23:18.275797 2120 catalog_manager.cc:4841] TS f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247): tablet 9fa293625be64799ab89f747048d5b7a (table test-table [id=182ff3b114ef435dbef289af5fbebdf4]) successfully deleted
I20260926 23:23:18.276120 2726 ts_tablet_manager.cc:1918] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:23:18.276260 2726 log.cc:1196] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/9fa293625be64799ab89f747048d5b7a
I20260926 23:23:18.276629 2726 ts_tablet_manager.cc:1939] T 9fa293625be64799ab89f747048d5b7a P 3311dc5268f44c8799ad3854f0abc741: Deleting consensus metadata
I20260926 23:23:18.277773 2123 catalog_manager.cc:4841] TS 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721): tablet 9fa293625be64799ab89f747048d5b7a (table test-table [id=182ff3b114ef435dbef289af5fbebdf4]) successfully deleted
I20260926 23:23:18.279250 2724 ts_tablet_manager.cc:1918] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:23:18.279394 2724 log.cc:1196] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/9fa293625be64799ab89f747048d5b7a
I20260926 23:23:18.280295 2724 ts_tablet_manager.cc:1939] T 9fa293625be64799ab89f747048d5b7a P a2db04d1d1424ce5854e926a813b6868: Deleting consensus metadata
I20260926 23:23:18.281289 2120 catalog_manager.cc:4841] TS a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193): tablet 9fa293625be64799ab89f747048d5b7a (table test-table [id=182ff3b114ef435dbef289af5fbebdf4]) successfully deleted
Sep 26 23:23:18 dist-test-slave-q5g8 krb5kdc[2091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:23:18 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:18.317859 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.304017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.0.97.62:42589)
0926 23:23:18.304635 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:18.304641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:18.304667 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:23:18.304857 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:18.304862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:18.304920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:18.305034 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:18.306290 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.307098 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.308370 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.308839 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.308989 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.312894 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:18.312899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:18.312902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:18.312932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:18.315400 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:18.315974 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:18.315978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:18.315981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:18.316086 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:18.316386 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:18.316388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:18.316390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:18.316615 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:18.316835 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:18.317057 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 23:23:18.317396 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":334,"threads_started":1}
I20260926 23:23:18.321756 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:18.322604 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:18.330051 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 997181ad697a4924933595b75681008f (DEFAULT_TABLE table=test-table [id=07a4622ef94c4c078f34c724d8f7a420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:18.330322 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 997181ad697a4924933595b75681008f (DEFAULT_TABLE table=test-table [id=07a4622ef94c4c078f34c724d8f7a420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:18.330561 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997181ad697a4924933595b75681008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:18.330742 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997181ad697a4924933595b75681008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:18.336073 2735 tablet_bootstrap.cc:492] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: Bootstrap starting.
I20260926 23:23:18.336306 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 997181ad697a4924933595b75681008f (DEFAULT_TABLE table=test-table [id=07a4622ef94c4c078f34c724d8f7a420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:18.336792 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997181ad697a4924933595b75681008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:18.337528 2736 tablet_bootstrap.cc:492] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: Bootstrap starting.
I20260926 23:23:18.338084 2735 tablet_bootstrap.cc:654] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:18.339234 2736 tablet_bootstrap.cc:654] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:18.341495 2736 tablet_bootstrap.cc:492] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: No bootstrap required, opened a new log
I20260926 23:23:18.341634 2736 ts_tablet_manager.cc:1397] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 23:23:18.342896 2735 tablet_bootstrap.cc:492] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: No bootstrap required, opened a new log
I20260926 23:23:18.342765 2736 raft_consensus.cc:348] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.342998 2736 raft_consensus.cc:374] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:18.343096 2736 raft_consensus.cc:729] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Initialized, Role: FOLLOWER
I20260926 23:23:18.343497 2735 ts_tablet_manager.cc:1397] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:23:18.343364 2736 consensus_queue.cc:260] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.343988 2736 ts_tablet_manager.cc:1428] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:18.344303 2735 raft_consensus.cc:348] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.344506 2735 raft_consensus.cc:374] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:18.344594 2735 raft_consensus.cc:729] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2c4e194a34a40829ae725df3db6c2, State: Initialized, Role: FOLLOWER
I20260926 23:23:18.344867 2735 consensus_queue.cc:260] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.345639 2735 ts_tablet_manager.cc:1428] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:18.349227 2739 tablet_bootstrap.cc:492] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: Bootstrap starting.
I20260926 23:23:18.350451 2739 tablet_bootstrap.cc:654] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:18.352100 2739 tablet_bootstrap.cc:492] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: No bootstrap required, opened a new log
I20260926 23:23:18.352340 2739 ts_tablet_manager.cc:1397] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 23:23:18.352953 2739 raft_consensus.cc:348] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.353169 2739 raft_consensus.cc:374] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:18.353282 2739 raft_consensus.cc:729] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3311dc5268f44c8799ad3854f0abc741, State: Initialized, Role: FOLLOWER
I20260926 23:23:18.353523 2739 consensus_queue.cc:260] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.353981 2739 ts_tablet_manager.cc:1428] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 23:23:18.354643 2705 raft_consensus.cc:482] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:18.354813 2705 raft_consensus.cc:504] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.355998 2705 leader_election.cc:288] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247), 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721)
I20260926 23:23:18.356405 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997181ad697a4924933595b75681008f" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2" is_pre_election: true
I20260926 23:23:18.356478 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997181ad697a4924933595b75681008f" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741" is_pre_election: true
I20260926 23:23:18.356621 2436 raft_consensus.cc:2454] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 0.
I20260926 23:23:18.356753 2571 raft_consensus.cc:2454] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 0.
I20260926 23:23:18.357090 2223 leader_election.cc:302] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2db04d1d1424ce5854e926a813b6868, f3c2c4e194a34a40829ae725df3db6c2; no voters:
I20260926 23:23:18.357426 2705 raft_consensus.cc:2781] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:18.357550 2705 raft_consensus.cc:482] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:18.357656 2705 raft_consensus.cc:3037] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:18.359616 2705 raft_consensus.cc:504] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.360046 2705 leader_election.cc:288] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 election: Requested vote from peers f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247), 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721)
I20260926 23:23:18.360549 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997181ad697a4924933595b75681008f" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c2c4e194a34a40829ae725df3db6c2"
I20260926 23:23:18.360700 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997181ad697a4924933595b75681008f" candidate_uuid: "a2db04d1d1424ce5854e926a813b6868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741"
I20260926 23:23:18.360781 2436 raft_consensus.cc:3037] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:18.360908 2571 raft_consensus.cc:3037] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:18.363703 2436 raft_consensus.cc:2454] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 1.
I20260926 23:23:18.363752 2571 raft_consensus.cc:2454] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2db04d1d1424ce5854e926a813b6868 in term 1.
I20260926 23:23:18.364185 2223 leader_election.cc:302] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2db04d1d1424ce5854e926a813b6868, f3c2c4e194a34a40829ae725df3db6c2; no voters:
I20260926 23:23:18.364619 2705 raft_consensus.cc:2781] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:18.364828 2705 raft_consensus.cc:686] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 1 LEADER]: Becoming Leader. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Running, Role: LEADER
I20260926 23:23:18.365175 2705 consensus_queue.cc:237] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.367575 2134 catalog_manager.cc:5495] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2db04d1d1424ce5854e926a813b6868 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a2db04d1d1424ce5854e926a813b6868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260926 23:23:18.387991 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.382646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.0.97.1:39193)
0926 23:23:18.383138 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:18.383142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:18.383161 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:23:18.383206 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:18.383209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:18.383288 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:18.383368 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:18.383664 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.385027 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.386346 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.386742 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.386878 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.387274 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:18.387376 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:18.387441 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:18.387573 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 23:23:18.387645 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":298,"threads_started":1}
I20260926 23:23:18.396261 2571 raft_consensus.cc:1261] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Refusing update from remote peer a2db04d1d1424ce5854e926a813b6868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:18.396273 2436 raft_consensus.cc:1261] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Refusing update from remote peer a2db04d1d1424ce5854e926a813b6868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:18.396854 2710 consensus_queue.cc:1035] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:18.397185 2683 consensus_queue.cc:1035] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:18.434840 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:18.435096 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:18.437997 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Sending DeleteTablet for 3 replicas of tablet 997181ad697a4924933595b75681008f
I20260926 23:23:18.438740 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 997181ad697a4924933595b75681008f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54402
I20260926 23:23:18.438802 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 997181ad697a4924933595b75681008f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260926 23:23:18.438802 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 997181ad697a4924933595b75681008f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50374
I20260926 23:23:18.438997 2724 tablet_replica.cc:330] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: stopping tablet replica
I20260926 23:23:18.439051 2726 tablet_replica.cc:330] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: stopping tablet replica
I20260926 23:23:18.439051 2725 tablet_replica.cc:330] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: stopping tablet replica
I20260926 23:23:18.439198 2724 raft_consensus.cc:2229] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:18.439240 2726 raft_consensus.cc:2229] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:18.439253 2725 raft_consensus.cc:2229] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:18.439410 2725 raft_consensus.cc:2258] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:18.439410 2726 raft_consensus.cc:2258] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:18.439481 2724 raft_consensus.cc:2258] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:18.440433 2725 ts_tablet_manager.cc:1905] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:18.440434 2726 ts_tablet_manager.cc:1905] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:18.440603 2724 ts_tablet_manager.cc:1905] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:18.440603 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:18.441393 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:18.446949 2725 ts_tablet_manager.cc:1918] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:18.446949 2726 ts_tablet_manager.cc:1918] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:18.447106 2725 log.cc:1196] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/997181ad697a4924933595b75681008f
I20260926 23:23:18.447105 2726 log.cc:1196] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/997181ad697a4924933595b75681008f
I20260926 23:23:18.447505 2726 ts_tablet_manager.cc:1939] T 997181ad697a4924933595b75681008f P 3311dc5268f44c8799ad3854f0abc741: Deleting consensus metadata
I20260926 23:23:18.447505 2725 ts_tablet_manager.cc:1939] T 997181ad697a4924933595b75681008f P f3c2c4e194a34a40829ae725df3db6c2: Deleting consensus metadata
I20260926 23:23:18.448576 2123 catalog_manager.cc:4841] TS 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721): tablet 997181ad697a4924933595b75681008f (table test-table [id=07a4622ef94c4c078f34c724d8f7a420]) successfully deleted
I20260926 23:23:18.448817 2120 catalog_manager.cc:4841] TS f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247): tablet 997181ad697a4924933595b75681008f (table test-table [id=07a4622ef94c4c078f34c724d8f7a420]) successfully deleted
I20260926 23:23:18.448987 2551 tablet_service.cc:1467] Processing CreateTablet for tablet be26e985ff3e4881a6d75df6cec73ecc (DEFAULT_TABLE table=test-table [id=ddc466d966e34c5685e6dbb83d15e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:18.449384 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be26e985ff3e4881a6d75df6cec73ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:18.449450 2412 tablet_service.cc:1467] Processing CreateTablet for tablet be26e985ff3e4881a6d75df6cec73ecc (DEFAULT_TABLE table=test-table [id=ddc466d966e34c5685e6dbb83d15e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:18.449558 2272 tablet_service.cc:1467] Processing CreateTablet for tablet be26e985ff3e4881a6d75df6cec73ecc (DEFAULT_TABLE table=test-table [id=ddc466d966e34c5685e6dbb83d15e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:18.449815 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be26e985ff3e4881a6d75df6cec73ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:18.449954 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be26e985ff3e4881a6d75df6cec73ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:18.450814 2724 ts_tablet_manager.cc:1918] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:23:18.451038 2724 log.cc:1196] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/997181ad697a4924933595b75681008f
I20260926 23:23:18.451599 2724 ts_tablet_manager.cc:1939] T 997181ad697a4924933595b75681008f P a2db04d1d1424ce5854e926a813b6868: Deleting consensus metadata
I20260926 23:23:18.452589 2120 catalog_manager.cc:4841] TS a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193): tablet 997181ad697a4924933595b75681008f (table test-table [id=07a4622ef94c4c078f34c724d8f7a420]) successfully deleted
I20260926 23:23:18.455475 2736 tablet_bootstrap.cc:492] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: Bootstrap starting.
I20260926 23:23:18.455622 2739 tablet_bootstrap.cc:492] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: Bootstrap starting.
I20260926 23:23:18.456775 2735 tablet_bootstrap.cc:492] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: Bootstrap starting.
I20260926 23:23:18.457535 2736 tablet_bootstrap.cc:654] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:18.457544 2739 tablet_bootstrap.cc:654] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:18.458042 2735 tablet_bootstrap.cc:654] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:18.459517 2739 tablet_bootstrap.cc:492] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: No bootstrap required, opened a new log
I20260926 23:23:18.459663 2739 ts_tablet_manager.cc:1397] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 23:23:18.459916 2736 tablet_bootstrap.cc:492] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: No bootstrap required, opened a new log
I20260926 23:23:18.460055 2736 ts_tablet_manager.cc:1397] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 23:23:18.460433 2735 tablet_bootstrap.cc:492] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: No bootstrap required, opened a new log
I20260926 23:23:18.460417 2739 raft_consensus.cc:348] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.460603 2735 ts_tablet_manager.cc:1397] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 23:23:18.460646 2739 raft_consensus.cc:374] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:18.460785 2739 raft_consensus.cc:729] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3311dc5268f44c8799ad3854f0abc741, State: Initialized, Role: FOLLOWER
I20260926 23:23:18.460772 2736 raft_consensus.cc:348] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.460999 2736 raft_consensus.cc:374] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:18.461310 2735 raft_consensus.cc:348] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.461522 2735 raft_consensus.cc:374] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:18.461014 2739 consensus_queue.cc:260] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.461516 2736 raft_consensus.cc:729] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2db04d1d1424ce5854e926a813b6868, State: Initialized, Role: FOLLOWER
I20260926 23:23:18.461591 2735 raft_consensus.cc:729] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c2c4e194a34a40829ae725df3db6c2, State: Initialized, Role: FOLLOWER
I20260926 23:23:18.461823 2735 consensus_queue.cc:260] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.461783 2736 consensus_queue.cc:260] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.461925 2739 ts_tablet_manager.cc:1428] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:18.462318 2735 ts_tablet_manager.cc:1428] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 23:23:18.463033 2736 ts_tablet_manager.cc:1428] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 23:23:18.502512 2740 raft_consensus.cc:482] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:18.502800 2740 raft_consensus.cc:504] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.504326 2740 leader_election.cc:288] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193), 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721)
I20260926 23:23:18.511236 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.504576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40721 (local address 127.0.97.2:38093)
0926 23:23:18.505218 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:18.505239 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:18.505405 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:18.505809 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:18.505814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:18.505870 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:18.506219 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.506232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.507568 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.507576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.509159 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.509170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.510376 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.510385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.510574 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.510620 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 23:23:18.510698 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:23:18.510789 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":40,"spinlock_wait_cycles":9216,"thread_start_us":310,"threads_started":1}
I20260926 23:23:18.511243 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.504989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33547 (local address 127.0.97.1:39193)
0926 23:23:18.505222 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:18.505229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:18.505827 (+ 598us) server_negotiation.cc:407] Connection header received
0926 23:23:18.506000 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:18.506004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:18.506058 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:18.506126 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:18.506794 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.507709 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.509571 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.510355 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.510479 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.510574 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:18.510889 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 23:23:18.511018 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 23:23:18.511360 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.504573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.97.2:33547)
0926 23:23:18.505696 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:18.505710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:18.505873 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:18.506173 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:18.506180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:18.506210 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:18.506616 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.506629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.507842 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.507847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.509420 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.509433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.510557 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.510561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.510715 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.510752 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:23:18.510842 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 23:23:18.510955 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":476,"spinlock_wait_cycles":16768,"thread_start_us":353,"threads_started":1}
I20260926 23:23:18.512195 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.504802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38093 (local address 127.0.97.3:40721)
0926 23:23:18.505508 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:18.505514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:18.505539 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:23:18.505612 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:18.505615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:18.505669 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:18.505757 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:18.506392 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.507380 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.509317 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.510196 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.510294 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.510374 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:18.510751 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 23:23:18.510899 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":427,"threads_started":1}
I20260926 23:23:18.512449 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be26e985ff3e4881a6d75df6cec73ecc" candidate_uuid: "f3c2c4e194a34a40829ae725df3db6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2db04d1d1424ce5854e926a813b6868" is_pre_election: true
I20260926 23:23:18.512746 2298 raft_consensus.cc:2454] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c2c4e194a34a40829ae725df3db6c2 in term 0.
I20260926 23:23:18.512861 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be26e985ff3e4881a6d75df6cec73ecc" candidate_uuid: "f3c2c4e194a34a40829ae725df3db6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741" is_pre_election: true
I20260926 23:23:18.513124 2571 raft_consensus.cc:2454] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c2c4e194a34a40829ae725df3db6c2 in term 0.
I20260926 23:23:18.513281 2363 leader_election.cc:302] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2db04d1d1424ce5854e926a813b6868, f3c2c4e194a34a40829ae725df3db6c2; no voters:
I20260926 23:23:18.513661 2740 raft_consensus.cc:2781] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:18.513796 2740 raft_consensus.cc:482] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:18.513911 2740 raft_consensus.cc:3037] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:18.515919 2740 raft_consensus.cc:504] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.516474 2740 leader_election.cc:288] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [CANDIDATE]: Term 1 election: Requested vote from peers a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193), 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721)
I20260926 23:23:18.516991 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be26e985ff3e4881a6d75df6cec73ecc" candidate_uuid: "f3c2c4e194a34a40829ae725df3db6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2db04d1d1424ce5854e926a813b6868"
I20260926 23:23:18.516975 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be26e985ff3e4881a6d75df6cec73ecc" candidate_uuid: "f3c2c4e194a34a40829ae725df3db6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3311dc5268f44c8799ad3854f0abc741"
I20260926 23:23:18.517179 2298 raft_consensus.cc:3037] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:18.517198 2571 raft_consensus.cc:3037] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:18.519986 2298 raft_consensus.cc:2454] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2c4e194a34a40829ae725df3db6c2 in term 1.
I20260926 23:23:18.519971 2571 raft_consensus.cc:2454] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c2c4e194a34a40829ae725df3db6c2 in term 1.
I20260926 23:23:18.520398 2366 leader_election.cc:302] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3311dc5268f44c8799ad3854f0abc741, f3c2c4e194a34a40829ae725df3db6c2; no voters:
I20260926 23:23:18.520790 2740 raft_consensus.cc:2781] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:18.521250 2740 raft_consensus.cc:686] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 1 LEADER]: Becoming Leader. State: Replica: f3c2c4e194a34a40829ae725df3db6c2, State: Running, Role: LEADER
I20260926 23:23:18.521612 2740 consensus_queue.cc:237] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } }
I20260926 23:23:18.524899 2136 catalog_manager.cc:5495] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f3c2c4e194a34a40829ae725df3db6c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c2c4e194a34a40829ae725df3db6c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260926 23:23:18.554067 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.548473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.0.97.2:42247)
0926 23:23:18.548994 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:18.548998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:18.549014 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:23:18.549085 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:18.549088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:18.549152 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:18.549320 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:18.549743 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.550475 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.551756 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.552188 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.552317 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.552837 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:18.553458 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:18.553634 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:18.553681 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 23:23:18.553764 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":293,"threads_started":1}
I20260926 23:23:18.569525 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.563393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44973 (local address 127.0.97.1:39193)
0926 23:23:18.563541 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:18.563545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:18.564120 (+ 575us) server_negotiation.cc:407] Connection header received
0926 23:23:18.564326 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:18.564335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:18.564398 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:18.564490 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:18.565485 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.566409 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.567721 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.568122 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.568269 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.568726 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:18.568894 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:18.568996 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:18.569139 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 23:23:18.569224 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 23:23:18.569584 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.563416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.97.2:44973)
0926 23:23:18.564023 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:18.564038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:18.564192 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:18.564514 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:18.564519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:18.564530 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:18.565328 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.565337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.566538 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.566543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.567577 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.567588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.568293 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.568299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.568435 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.568505 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:18.569009 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:18.569020 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:23:18.569103 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 23:23:18.569236 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":314,"threads_started":1}
I20260926 23:23:18.581387 2571 raft_consensus.cc:1261] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Refusing update from remote peer f3c2c4e194a34a40829ae725df3db6c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:18.581890 2298 raft_consensus.cc:1261] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Refusing update from remote peer f3c2c4e194a34a40829ae725df3db6c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:18.582544 2754 consensus_queue.cc:1035] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3311dc5268f44c8799ad3854f0abc741" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 23:23:18.583683 2754 consensus_queue.cc:1035] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2db04d1d1424ce5854e926a813b6868" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:18.631143 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.625850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34987 (local address 127.0.97.2:42247)
0926 23:23:18.626014 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:18.626018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:18.626864 (+ 846us) server_negotiation.cc:407] Connection header received
0926 23:23:18.627040 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:18.627047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:18.627136 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:18.627255 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:18.627724 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.628402 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.629770 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.630152 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.630244 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.630596 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:18.630702 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:18.630765 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:18.630888 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 23:23:18.630943 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:23:18.631268 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:18.625783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42247 (local address 127.0.97.1:34987)
0926 23:23:18.626756 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:18.626771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:18.626911 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:18.627292 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:18.627296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:18.627310 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:23:18.627589 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.627597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.628512 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.628516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.629652 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:18.629663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:18.630313 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:18.630317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:18.630434 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:18.630468 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:23:18.630785 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:23:18.630793 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:23:18.630850 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 23:23:18.630941 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":694,"threads_started":1}
W20260926 23:23:18.632201 2707 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 23:23:20.252233 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:20.252516 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:20.256907 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3adafb1f74f388afc630d73a30170: Sending DeleteTablet for 3 replicas of tablet be26e985ff3e4881a6d75df6cec73ecc
I20260926 23:23:20.257658 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet be26e985ff3e4881a6d75df6cec73ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54402
I20260926 23:23:20.257699 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet be26e985ff3e4881a6d75df6cec73ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260926 23:23:20.257750 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet be26e985ff3e4881a6d75df6cec73ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50374
I20260926 23:23:20.258339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2207
I20260926 23:23:20.270256 2789 tablet_replica.cc:330] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: stopping tablet replica
I20260926 23:23:20.270272 2790 tablet_replica.cc:330] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: stopping tablet replica
I20260926 23:23:20.270547 2790 raft_consensus.cc:2229] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:20.270573 2789 raft_consensus.cc:2229] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:20.270962 2790 raft_consensus.cc:2258] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:20.271340 2789 raft_consensus.cc:2258] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:20.272872 2789 ts_tablet_manager.cc:1905] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:20.273002 2790 ts_tablet_manager.cc:1905] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:20.278517 2791 tablet_replica.cc:330] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: stopping tablet replica
I20260926 23:23:20.278719 2791 raft_consensus.cc:2229] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:20.278970 2791 raft_consensus.cc:2258] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:20.279799 2791 ts_tablet_manager.cc:1905] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:20.291347 2791 ts_tablet_manager.cc:1918] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:23:20.291381 2789 ts_tablet_manager.cc:1918] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:23:20.291522 2789 log.cc:1196] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/be26e985ff3e4881a6d75df6cec73ecc
I20260926 23:23:20.291569 2791 log.cc:1196] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/be26e985ff3e4881a6d75df6cec73ecc
I20260926 23:23:20.291872 2790 ts_tablet_manager.cc:1918] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:23:20.292049 2790 log.cc:1196] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/be26e985ff3e4881a6d75df6cec73ecc
I20260926 23:23:20.292117 2791 ts_tablet_manager.cc:1939] T be26e985ff3e4881a6d75df6cec73ecc P 3311dc5268f44c8799ad3854f0abc741: Deleting consensus metadata
I20260926 23:23:20.292157 2789 ts_tablet_manager.cc:1939] T be26e985ff3e4881a6d75df6cec73ecc P f3c2c4e194a34a40829ae725df3db6c2: Deleting consensus metadata
I20260926 23:23:20.292547 2790 ts_tablet_manager.cc:1939] T be26e985ff3e4881a6d75df6cec73ecc P a2db04d1d1424ce5854e926a813b6868: Deleting consensus metadata
Sep 26 23:23:20 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:20.302357 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:20.260076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.0.97.1:39193)
0926 23:23:20.278287 (+ 18211us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:20.278294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:20.278319 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:23:20.278377 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:20.278382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:20.278454 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:20.278577 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:20.280219 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:20.281339 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:20.282588 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:20.282999 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:20.283094 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:20.296469 (+ 13375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:20.296484 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:20.296501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:20.296543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:20.299500 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:20.300170 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:20.300177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:20.300179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:20.300302 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:20.300784 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:20.300788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:20.300790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:20.301153 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:20.301404 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:20.301726 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:23:20.301988 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18081,"thread_start_us":17967,"threads_started":1}
W20260926 23:23:20.303175 2338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50264
W20260926 23:23:20.303893 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:20.304159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 2207
I20260926 23:23:20.342479 2123 catalog_manager.cc:4841] TS 3311dc5268f44c8799ad3854f0abc741 (127.0.97.3:40721): tablet be26e985ff3e4881a6d75df6cec73ecc (table test-table [id=ddc466d966e34c5685e6dbb83d15e171]) successfully deleted
I20260926 23:23:20.342480 2120 catalog_manager.cc:4841] TS f3c2c4e194a34a40829ae725df3db6c2 (127.0.97.2:42247): tablet be26e985ff3e4881a6d75df6cec73ecc (table test-table [id=ddc466d966e34c5685e6dbb83d15e171]) successfully deleted
W20260926 23:23:20.381600 2120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39193 (error 108)
I20260926 23:23:20.381997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2347
I20260926 23:23:20.383956 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:20.383118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:50274)
0926 23:23:20.383638 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:20.383754 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":256,"threads_started":1}
W20260926 23:23:20.384209 2120 catalog_manager.cc:4584] TS a2db04d1d1424ce5854e926a813b6868 (127.0.97.1:39193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be26e985ff3e4881a6d75df6cec73ecc: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39193: connect: Connection refused (error 111)
Sep 26 23:23:20 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:20.397562 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:20.382786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.0.97.2:42247)
0926 23:23:20.383513 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:20.383519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:20.383538 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:23:20.383621 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:20.383625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:20.383680 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:20.383797 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:20.385214 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:20.385960 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:20.387158 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:20.387551 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:20.387689 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:20.391788 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:20.391800 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:20.391816 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:20.391854 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:20.394853 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:20.395480 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:20.395491 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:20.395499 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:20.395636 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:20.395979 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:20.395985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:20.395989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:20.396328 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:20.396656 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:20.396953 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 23:23:20.397167 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":467,"threads_started":1}
W20260926 23:23:20.398286 2477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52158
W20260926 23:23:20.398952 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:20.399106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 2347
I20260926 23:23:20.421341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2486
Sep 26 23:23:20 dist-test-slave-q5g8 krb5kdc[2091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790464998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:20.436848 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:20.422149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.0.97.3:40721)
0926 23:23:20.422675 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:20.422679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:20.422695 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:23:20.422747 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:20.422750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:20.422804 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:20.422890 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:20.424147 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:20.424935 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:20.426245 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:20.426616 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:20.426710 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:20.431101 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:20.431115 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:20.431132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:20.431163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:20.433931 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:20.434702 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:20.434711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:20.434713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:20.434854 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:20.435207 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:20.435211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:20.435213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:20.435586 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:20.435833 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:20.436185 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 23:23:20.436464 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":303,"threads_started":1}
W20260926 23:23:20.437561 2616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54476
I20260926 23:23:20.438266 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:20.437817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:50290)
0926 23:23:20.438007 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:20.438111 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 23:23:20.438257 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2486: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:20.438458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 2486
I20260926 23:23:20.439764 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:20.439351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:50300)
0926 23:23:20.439519 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:20.439626 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 23:23:20.465036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2104
I20260926 23:23:20.466244 2176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:20.580528 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:20.579981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:50302)
0926 23:23:20.580175 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:20.580315 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 23:23:20.580996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 2104
2026-09-26T23:23:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8336 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:20 dist-test-slave-q5g8 krb5kdc[2801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:20 dist-test-slave-q5g8 krb5kdc[2801](info): set up 2 sockets
Sep 26 23:23:20 dist-test-slave-q5g8 krb5kdc[2801](info): commencing operation
krb5kdc: starting...
W20260926 23:23:21.669131 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:21 dist-test-slave-q5g8 krb5kdc[2801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:21.894212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39643
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:22.257426 2814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:22.257747 2814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:22.257838 2814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:22.265388 2814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:22.265511 2814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:22.265563 2814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:22.265611 2814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:22.265656 2814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:22.275987 2814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:22.277372 2814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:22.278676 2814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:22.292214 2822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:22.292248 2820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:22.292320 2819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:22.293473 2814 server_base.cc:1034] running on GCE node
I20260926 23:23:22.294539 2814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:22.296408 2814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:22.297727 2814 hybrid_clock.cc:648] HybridClock initialized: now 1790465002297635 us; error 101 us; skew 500 ppm
I20260926 23:23:22.298219 2814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:22 dist-test-slave-q5g8 krb5kdc[2801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:22.301795 2814 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:22.303423 2814 webserver.cc:460] Webserver started at http://127.0.97.62:34967/ using document root <none> and password file <none>
I20260926 23:23:22.304240 2814 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:22.304430 2814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:22.304848 2814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:22.308027 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "bf975daf47a94bd39a626b5ed9922b45"
format_stamp: "Formatted at 2026-09-26 23:23:22 on dist-test-slave-q5g8"
server_key: "a00bcce9767516c85815f725fab83a49"
server_key_iv: "68bbf3500abfeb1ce968f6f6c2aa3781"
server_key_version: "encryptionkey@0"
I20260926 23:23:22.308983 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "bf975daf47a94bd39a626b5ed9922b45"
format_stamp: "Formatted at 2026-09-26 23:23:22 on dist-test-slave-q5g8"
server_key: "a00bcce9767516c85815f725fab83a49"
server_key_iv: "68bbf3500abfeb1ce968f6f6c2aa3781"
server_key_version: "encryptionkey@0"
I20260926 23:23:22.314656 2814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:23:22.319586 2829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:22.321980 2814 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 23:23:22.322264 2814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "bf975daf47a94bd39a626b5ed9922b45"
format_stamp: "Formatted at 2026-09-26 23:23:22 on dist-test-slave-q5g8"
server_key: "a00bcce9767516c85815f725fab83a49"
server_key_iv: "68bbf3500abfeb1ce968f6f6c2aa3781"
server_key_version: "encryptionkey@0"
I20260926 23:23:22.322604 2814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:22.341012 2814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:22.346108 2814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:22.346381 2814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:22.367081 2814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39643
I20260926 23:23:22.367111 2880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39643 every 8 connection(s)
I20260926 23:23:22.368661 2814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:22.373081 2881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:22.375638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 2814
I20260926 23:23:22.375908 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:22.376793 388 external_mini_cluster.cc:1428] Setting key 8a21e6c35c5f3ce2723fdd0fd0921063
I20260926 23:23:22.387142 2881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45: Bootstrap starting.
I20260926 23:23:22.390853 2881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:22.391997 2881 log.cc:824] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:23:22 dist-test-slave-q5g8 krb5kdc[2801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:22.395565 2881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45: No bootstrap required, opened a new log
I20260926 23:23:22.402149 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:22.379402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.0.97.62:39643)
0926 23:23:22.380429 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:22.380448 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:22.380502 (+ 54us) server_negotiation.cc:407] Connection header received
0926 23:23:22.382563 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:22.382592 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:22.383089 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:22.383580 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:22.385354 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:22.386779 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:22.388299 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:22.389199 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:22.389439 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:22.394078 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:22.394087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:22.394106 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:22.394144 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:22.397545 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:22.398415 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:22.398423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:22.398430 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:22.398610 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:22.399045 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:22.399050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:22.399053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:22.399664 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:22.400002 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:22.400487 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 23:23:22.400982 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":476,"threads_started":1}
I20260926 23:23:22.404597 2881 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf975daf47a94bd39a626b5ed9922b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39643 } }
I20260926 23:23:22.405124 2881 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:22.405243 2881 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf975daf47a94bd39a626b5ed9922b45, State: Initialized, Role: FOLLOWER
I20260926 23:23:22.405963 2881 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf975daf47a94bd39a626b5ed9922b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39643 } }
I20260926 23:23:22.406236 2881 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:22.406415 2881 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:22.406603 2881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:22.408979 2881 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf975daf47a94bd39a626b5ed9922b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39643 } }
I20260926 23:23:22.409562 2881 leader_election.cc:302] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf975daf47a94bd39a626b5ed9922b45; no voters:
I20260926 23:23:22.410274 2881 leader_election.cc:288] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:22.410440 2886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:22.411351 2886 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [term 1 LEADER]: Becoming Leader. State: Replica: bf975daf47a94bd39a626b5ed9922b45, State: Running, Role: LEADER
I20260926 23:23:22.412158 2886 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf975daf47a94bd39a626b5ed9922b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39643 } }
I20260926 23:23:22.413022 2881 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:22.415834 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf975daf47a94bd39a626b5ed9922b45. Latest consensus state: current_term: 1 leader_uuid: "bf975daf47a94bd39a626b5ed9922b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf975daf47a94bd39a626b5ed9922b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39643 } } }
I20260926 23:23:22.416275 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:22.416909 2887 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf975daf47a94bd39a626b5ed9922b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf975daf47a94bd39a626b5ed9922b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39643 } } }
I20260926 23:23:22.417222 2887 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:22.418999 2894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:22.425449 2894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:22.436554 2894 catalog_manager.cc:1347] Generated new cluster ID: 3c6cd34c2df24d0a938a140200e52305
I20260926 23:23:22.436712 2894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:22.469425 2894 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:22.470971 2894 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:22.482185 2894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf975daf47a94bd39a626b5ed9922b45: Generated new TSK 0
I20260926 23:23:22.483424 2894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:22.542003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39643
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:22.891350 2907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:22.891652 2907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:22.891817 2907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:22.899421 2907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:22.899545 2907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:22.899727 2907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:22.910166 2907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:22.911558 2907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:22.913079 2907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:22.928575 2912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:22.928704 2913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:22.928601 2915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:22.930145 2907 server_base.cc:1034] running on GCE node
I20260926 23:23:22.930972 2907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:22.932044 2907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:22.933297 2907 hybrid_clock.cc:648] HybridClock initialized: now 1790465002933235 us; error 51 us; skew 500 ppm
I20260926 23:23:22.933676 2907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:22 dist-test-slave-q5g8 krb5kdc[2801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:22.936959 2907 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:22.938436 2907 webserver.cc:460] Webserver started at http://127.0.97.1:42311/ using document root <none> and password file <none>
I20260926 23:23:22.939111 2907 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:22.939213 2907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:22.939474 2907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:22.942792 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "14977a8d431f4eb9bc7e679c3cb36843"
format_stamp: "Formatted at 2026-09-26 23:23:22 on dist-test-slave-q5g8"
server_key: "8776c54dba34aeaf5d02f7dff5c21f8b"
server_key_iv: "a750d24fbc4406483ee4a1c40867c868"
server_key_version: "encryptionkey@0"
I20260926 23:23:22.943602 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "14977a8d431f4eb9bc7e679c3cb36843"
format_stamp: "Formatted at 2026-09-26 23:23:22 on dist-test-slave-q5g8"
server_key: "8776c54dba34aeaf5d02f7dff5c21f8b"
server_key_iv: "a750d24fbc4406483ee4a1c40867c868"
server_key_version: "encryptionkey@0"
I20260926 23:23:22.949237 2907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:23:22.952992 2922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:22.954527 2907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:22.954723 2907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "14977a8d431f4eb9bc7e679c3cb36843"
format_stamp: "Formatted at 2026-09-26 23:23:22 on dist-test-slave-q5g8"
server_key: "8776c54dba34aeaf5d02f7dff5c21f8b"
server_key_iv: "a750d24fbc4406483ee4a1c40867c868"
server_key_version: "encryptionkey@0"
I20260926 23:23:22.954932 2907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:22.976964 2907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:22.981426 2907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:22.981787 2907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:22.983004 2907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:22.984392 2907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:22.984468 2907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:22.984640 2907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:22.984748 2907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 23:23:23.022249 2907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36445
I20260926 23:23:23.022349 3034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36445 every 8 connection(s)
I20260926 23:23:23.023592 2907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:23.033025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 2907
I20260926 23:23:23.033273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:23.034052 388 external_mini_cluster.cc:1428] Setting key ad5cef67901e84857728ddf5dfe835a1
Sep 26 23:23:23 dist-test-slave-q5g8 krb5kdc[2801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:23.047464 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:23.026302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41545 (local address 127.0.97.62:39643)
0926 23:23:23.026782 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:23.026787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:23.027766 (+ 979us) server_negotiation.cc:407] Connection header received
0926 23:23:23.029902 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:23.029907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:23.030008 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:23.030141 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:23.031728 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.032504 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.035088 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.035529 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.035687 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:23.040373 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:23.040379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:23.040382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:23.040420 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:23.043300 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:23.044314 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:23.044320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:23.044322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:23.044464 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:23.045099 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:23.045108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:23.045123 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:23.045447 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:23.045682 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:23.046638 (+ 956us) server_negotiation.cc:299] Negotiation successful
0926 23:23:23.046967 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260926 23:23:23.048875 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:23.026775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39643 (local address 127.0.97.1:41545)
0926 23:23:23.027525 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:23.027574 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:23.029584 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:23.030403 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:23.030413 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:23.030954 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:23.031531 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:23.031547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.032751 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.032759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:23.034905 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:23.034920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.035787 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.035796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:23.036088 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:23.037056 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:23.037094 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:23.040048 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:23.043523 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:23.043545 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:23.043564 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:23.044120 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:23.044614 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:23.044620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:23.044624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:23.044943 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:23.045640 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:23.045657 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:23.046126 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 23:23:23.046498 (+ 372us) client_negotiation.cc:240] Negotiation successful
0926 23:23:23.047232 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":306,"threads_started":1}
I20260926 23:23:23.051317 3035 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39643
I20260926 23:23:23.051759 3035 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:23.052680 3035 heartbeater.cc:511] Master 127.0.97.62:39643 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:23:23.055346 2846 ts_manager.cc:194] Registered new tserver with Master: 14977a8d431f4eb9bc7e679c3cb36843 (127.0.97.1:36445)
I20260926 23:23:23.057006 2846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:23.086440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39643
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:23.441157 3041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:23.441524 3041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:23.441707 3041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:23.449100 3041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:23.449292 3041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:23.449507 3041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:23.460032 3041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:23.461452 3041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:23.462901 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:23.477239 3049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:23.477298 3046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:23.477329 3047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:23.478463 3041 server_base.cc:1034] running on GCE node
I20260926 23:23:23.479094 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:23.480156 3041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:23.481374 3041 hybrid_clock.cc:648] HybridClock initialized: now 1790465003481298 us; error 61 us; skew 500 ppm
I20260926 23:23:23.481745 3041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:23 dist-test-slave-q5g8 krb5kdc[2801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:23.485435 3041 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:23.487138 3041 webserver.cc:460] Webserver started at http://127.0.97.2:36903/ using document root <none> and password file <none>
I20260926 23:23:23.487922 3041 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:23.488061 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:23.488407 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:23.491541 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "a3aca10a311d40b9855cecf1721f3b7e"
format_stamp: "Formatted at 2026-09-26 23:23:23 on dist-test-slave-q5g8"
server_key: "aa0d3674a5a045574b361c2b54dfa4de"
server_key_iv: "7e6580e760c6baa203c89231c3f7216b"
server_key_version: "encryptionkey@0"
I20260926 23:23:23.492386 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a3aca10a311d40b9855cecf1721f3b7e"
format_stamp: "Formatted at 2026-09-26 23:23:23 on dist-test-slave-q5g8"
server_key: "aa0d3674a5a045574b361c2b54dfa4de"
server_key_iv: "7e6580e760c6baa203c89231c3f7216b"
server_key_version: "encryptionkey@0"
I20260926 23:23:23.498175 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 23:23:23.502327 3056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:23.504014 3041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:23:23.504245 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "a3aca10a311d40b9855cecf1721f3b7e"
format_stamp: "Formatted at 2026-09-26 23:23:23 on dist-test-slave-q5g8"
server_key: "aa0d3674a5a045574b361c2b54dfa4de"
server_key_iv: "7e6580e760c6baa203c89231c3f7216b"
server_key_version: "encryptionkey@0"
I20260926 23:23:23.504491 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:23.523303 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:23.527756 3041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:23.528052 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:23.529150 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:23.530618 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:23.530753 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:23.530889 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:23.530962 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:23.564532 3041 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45541
I20260926 23:23:23.564577 3168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45541 every 8 connection(s)
I20260926 23:23:23.566076 3041 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:23.567061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3041
I20260926 23:23:23.567289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:23.567853 388 external_mini_cluster.cc:1428] Setting key 80271c5e8f8a6f7d611c36017ef58ef4
Sep 26 23:23:23 dist-test-slave-q5g8 krb5kdc[2801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:23:23.595340 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:23.569921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34111 (local address 127.0.97.62:39643)
0926 23:23:23.570491 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:23.570499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:23.571880 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 23:23:23.575663 (+ 3783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:23.575669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:23.575792 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:23.575933 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:23.578724 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.579923 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.583132 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.583587 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.583699 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:23.588894 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:23.588900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:23.588904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:23.588943 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:23.591620 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:23.592572 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:23.592577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:23.592580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:23.592781 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:23.593379 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:23.593385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:23.593388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:23.593680 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:23.593878 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:23.594720 (+ 842us) server_negotiation.cc:299] Negotiation successful
0926 23:23:23.594958 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
I20260926 23:23:23.596496 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:23.570658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39643 (local address 127.0.97.2:34111)
0926 23:23:23.571642 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:23.571711 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:23.575191 (+ 3480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:23.576343 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:23.576362 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:23.577521 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:23.578456 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:23.578481 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.580145 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.580155 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:23.582907 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:23.582932 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:23.583878 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:23.583889 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:23.584318 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:23.585262 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:23.585301 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:23.588559 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:23.591826 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:23.591837 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:23.591858 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:23.592403 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:23.592905 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:23.592910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:23.592913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:23.593242 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:23.593830 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:23.593840 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:23.594296 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 23:23:23.594615 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 23:23:23.595217 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":368,"threads_started":1}
I20260926 23:23:23.598305 3169 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39643
I20260926 23:23:23.598665 3169 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:23.599485 3169 heartbeater.cc:511] Master 127.0.97.62:39643 requested a full tablet report, sending...
I20260926 23:23:23.601109 2846 ts_manager.cc:194] Registered new tserver with Master: a3aca10a311d40b9855cecf1721f3b7e (127.0.97.2:45541)
I20260926 23:23:23.601958 2846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:23.624799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39643
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:23.971277 3175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:23.971652 3175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:23.971848 3175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:23.979122 3175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:23.979307 3175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:23.979509 3175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:23.989429 3175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:23.990847 3175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:23.992246 3175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:24.005652 3183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:24.005625 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:24.005944 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:24.006451 3175 server_base.cc:1034] running on GCE node
I20260926 23:23:24.007138 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:24.008366 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:24.009560 3175 hybrid_clock.cc:648] HybridClock initialized: now 1790465004009505 us; error 52 us; skew 500 ppm
I20260926 23:23:24.009899 3175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:24 dist-test-slave-q5g8 krb5kdc[2801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:24.013080 3175 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:24.014828 3175 webserver.cc:460] Webserver started at http://127.0.97.3:37123/ using document root <none> and password file <none>
I20260926 23:23:24.015532 3175 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:24.015633 3175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:24.015956 3175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:24.019028 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "7c0529c37467413a81aa7f0f989d42ea"
format_stamp: "Formatted at 2026-09-26 23:23:24 on dist-test-slave-q5g8"
server_key: "7e6bfb5ad2dc3c2518699ea342e205f2"
server_key_iv: "a69e516cd9d4c52ed996845e12ae09cc"
server_key_version: "encryptionkey@0"
I20260926 23:23:24.019902 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7c0529c37467413a81aa7f0f989d42ea"
format_stamp: "Formatted at 2026-09-26 23:23:24 on dist-test-slave-q5g8"
server_key: "7e6bfb5ad2dc3c2518699ea342e205f2"
server_key_iv: "a69e516cd9d4c52ed996845e12ae09cc"
server_key_version: "encryptionkey@0"
I20260926 23:23:24.025800 3175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 23:23:24.029747 3190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:24.031247 3175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 23:23:24.031474 3175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "7c0529c37467413a81aa7f0f989d42ea"
format_stamp: "Formatted at 2026-09-26 23:23:24 on dist-test-slave-q5g8"
server_key: "7e6bfb5ad2dc3c2518699ea342e205f2"
server_key_iv: "a69e516cd9d4c52ed996845e12ae09cc"
server_key_version: "encryptionkey@0"
I20260926 23:23:24.031816 3175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:24.061455 3035 heartbeater.cc:503] Master 127.0.97.62:39643 was elected leader, sending a full tablet report...
I20260926 23:23:24.087927 3175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:24.093464 3175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:24.093869 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:24.095278 3175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:24.097115 3175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:24.097216 3175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:24.097450 3175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:24.097536 3175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:24.137415 3175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46735
I20260926 23:23:24.137465 3302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46735 every 8 connection(s)
I20260926 23:23:24.139084 3175 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:24.146370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3175
I20260926 23:23:24.146622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:24.147105 388 external_mini_cluster.cc:1428] Setting key 5441d170f8f6160f3243b48968c82fd8
Sep 26 23:23:24 dist-test-slave-q5g8 krb5kdc[2801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:24.164897 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:24.142151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39061 (local address 127.0.97.62:39643)
0926 23:23:24.142679 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:24.142686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:24.143971 (+ 1285us) server_negotiation.cc:407] Connection header received
0926 23:23:24.146070 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:24.146079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:24.146186 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:24.146339 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:24.148540 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:24.149762 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:24.152223 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:24.152809 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:24.152937 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:24.157192 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:24.157198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:24.157201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:24.157243 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:24.160483 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:24.161571 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:24.161577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:24.161591 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:24.161785 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:24.162406 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:24.162413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:24.162417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:24.162924 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:24.163105 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:24.164164 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0926 23:23:24.164457 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":250,"threads_started":1}
I20260926 23:23:24.165918 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:24.142866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39643 (local address 127.0.97.3:39061)
0926 23:23:24.143762 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:24.143813 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:24.145715 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:24.146793 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:24.146805 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:24.147522 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:24.148298 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:24.148329 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:24.149995 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:24.150004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:24.152023 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:24.152039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:24.153058 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:24.153064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:24.153322 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:24.154100 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:24.154128 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:24.156875 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:24.160837 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:24.160856 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:24.160873 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:24.161390 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:24.161955 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:24.161960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:24.161963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:24.162250 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:24.163077 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:24.163084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:24.163565 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 23:23:24.163991 (+ 426us) client_negotiation.cc:240] Negotiation successful
0926 23:23:24.164557 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":345,"threads_started":1}
I20260926 23:23:24.167696 3303 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39643
I20260926 23:23:24.168033 3303 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:24.168953 3303 heartbeater.cc:511] Master 127.0.97.62:39643 requested a full tablet report, sending...
I20260926 23:23:24.170542 2846 ts_manager.cc:194] Registered new tserver with Master: 7c0529c37467413a81aa7f0f989d42ea (127.0.97.3:46735)
I20260926 23:23:24.171429 2846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39061
I20260926 23:23:24.177820 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:23:24.210677 3305 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:24.208062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.0.97.62:39643)
0926 23:23:24.208297 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:24.208305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:24.208861 (+ 556us) server_negotiation.cc:407] Connection header received
0926 23:23:24.209058 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:24.209062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:24.209127 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:24.209241 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:24.210488 (+ 1247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58506: BlockingRecv error: recv got EOF from 127.0.0.1:58506 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:23:24.214646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2907
W20260926 23:23:24.218034 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2907: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:36445: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:23:24.218218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 2907
I20260926 23:23:24.237022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3041
W20260926 23:23:24.239763 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3041: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45541: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:23:24.239945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3041
I20260926 23:23:24.257221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3175
W20260926 23:23:24.260783 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3175: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46735: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:23:24.260936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3175
I20260926 23:23:24.273962 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 2814
I20260926 23:23:24.275511 2876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:24.388249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 2814
2026-09-26T23:23:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3801 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:24 dist-test-slave-q5g8 krb5kdc[3321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:24 dist-test-slave-q5g8 krb5kdc[3321](info): set up 2 sockets
Sep 26 23:23:24 dist-test-slave-q5g8 krb5kdc[3321](info): commencing operation
krb5kdc: starting...
W20260926 23:23:25.501179 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:25 dist-test-slave-q5g8 krb5kdc[3321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260926 23:23:26.516664 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.841s user 0.005s sys 0.005s
I20260926 23:23:26.518278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42559
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:26.893501 3334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:26.893929 3334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:26.894114 3334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:26.902264 3334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:26.902478 3334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:26.902602 3334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:26.902716 3334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:26.902819 3334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:26.914320 3334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:26.915762 3334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:26.917268 3334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:26.930397 3339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:26.930377 3340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:26.930498 3342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:26.931340 3334 server_base.cc:1034] running on GCE node
I20260926 23:23:26.932319 3334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:26.934151 3334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:26.935504 3334 hybrid_clock.cc:648] HybridClock initialized: now 1790465006935454 us; error 63 us; skew 500 ppm
I20260926 23:23:26.935959 3334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:26 dist-test-slave-q5g8 krb5kdc[3321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:26.946986 3334 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:26.948850 3334 webserver.cc:460] Webserver started at http://127.0.97.62:42833/ using document root <none> and password file <none>
I20260926 23:23:26.949622 3334 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:26.949795 3334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:26.950129 3334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:26.953521 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "2e92bdc5ea5240cf8993e785f2ca2e72"
format_stamp: "Formatted at 2026-09-26 23:23:26 on dist-test-slave-q5g8"
server_key: "962077c20790d66bc8dbff8532f6c757"
server_key_iv: "4332d7d7f6c3631270e1b44025ad1d32"
server_key_version: "encryptionkey@0"
I20260926 23:23:26.954460 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "2e92bdc5ea5240cf8993e785f2ca2e72"
format_stamp: "Formatted at 2026-09-26 23:23:26 on dist-test-slave-q5g8"
server_key: "962077c20790d66bc8dbff8532f6c757"
server_key_iv: "4332d7d7f6c3631270e1b44025ad1d32"
server_key_version: "encryptionkey@0"
I20260926 23:23:26.968417 3334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:23:26.972429 3349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:26.974051 3334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:26.974306 3334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "2e92bdc5ea5240cf8993e785f2ca2e72"
format_stamp: "Formatted at 2026-09-26 23:23:26 on dist-test-slave-q5g8"
server_key: "962077c20790d66bc8dbff8532f6c757"
server_key_iv: "4332d7d7f6c3631270e1b44025ad1d32"
server_key_version: "encryptionkey@0"
I20260926 23:23:26.974553 3334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:26.997236 3334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:27.001901 3334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:27.002265 3334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:27.026032 3334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42559
I20260926 23:23:27.026036 3400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42559 every 8 connection(s)
I20260926 23:23:27.027681 3334 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:27.031953 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:27.032472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3334
I20260926 23:23:27.032742 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:27.033586 388 external_mini_cluster.cc:1428] Setting key bc0a5de82dbafc41e2f1d5af18dced7d
I20260926 23:23:27.049064 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72: Bootstrap starting.
Sep 26 23:23:27 dist-test-slave-q5g8 krb5kdc[3321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:27.053449 3401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:27.054816 3401 log.cc:824] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:27.058264 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72: No bootstrap required, opened a new log
I20260926 23:23:27.061417 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:27.035995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.0.97.62:42559)
0926 23:23:27.037112 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:27.037127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:27.037182 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:23:27.039425 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:27.039458 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:27.040183 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:27.040657 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:27.043018 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:27.044779 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:27.046847 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:27.047753 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:27.047956 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:27.053125 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:27.053133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:27.053152 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:27.053198 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:27.056635 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:27.057546 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:27.057555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:27.057563 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:27.057744 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:27.058198 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:27.058204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:27.058207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:27.058851 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:27.059201 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:27.059676 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 23:23:27.060180 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":511,"threads_started":1}
I20260926 23:23:27.066145 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:23:27.067694 3401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260926 23:23:27.068117 3401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:27.068233 3401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e92bdc5ea5240cf8993e785f2ca2e72, State: Initialized, Role: FOLLOWER
I20260926 23:23:27.068948 3401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260926 23:23:27.069216 3401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:27.069363 3401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:27.069595 3401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:27.071852 3401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260926 23:23:27.072428 3401 leader_election.cc:302] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e92bdc5ea5240cf8993e785f2ca2e72; no voters:
I20260926 23:23:27.073217 3401 leader_election.cc:288] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:27.073329 3407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:27.074607 3407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [term 1 LEADER]: Becoming Leader. State: Replica: 2e92bdc5ea5240cf8993e785f2ca2e72, State: Running, Role: LEADER
I20260926 23:23:27.075304 3407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260926 23:23:27.075875 3401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:27.078348 3409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e92bdc5ea5240cf8993e785f2ca2e72. Latest consensus state: current_term: 1 leader_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } } }
I20260926 23:23:27.078795 3409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:27.078897 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e92bdc5ea5240cf8993e785f2ca2e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } } }
I20260926 23:23:27.079373 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:27.088512 3416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:27.094095 3416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:27.111972 3416 catalog_manager.cc:1347] Generated new cluster ID: 436cf5bc25c845f4bb835a1ef88ea050
I20260926 23:23:27.112185 3416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:27.124871 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:27.115281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.0.97.62:42559)
0926 23:23:27.115467 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:27.115474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:27.116045 (+ 571us) server_negotiation.cc:407] Connection header received
0926 23:23:27.116225 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:27.116231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:27.116319 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:27.116441 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:27.117659 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:27.118807 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:27.120721 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:27.121467 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:27.121614 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:27.122985 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:27.122993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:27.123020 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:27.123962 (+ 942us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:27.124046 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:27.124075 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:27.124188 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:27.124362 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 23:23:27.124468 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260926 23:23:27.126009 3416 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:27.127971 3416 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:27.136723 3416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e92bdc5ea5240cf8993e785f2ca2e72: Generated new TSK 0
I20260926 23:23:27.137694 3416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:23:27.148515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3334
I20260926 23:23:27.149866 3396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:27.244935 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3334
2026-09-26T23:23:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2856 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:27 dist-test-slave-q5g8 krb5kdc[3435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:27 dist-test-slave-q5g8 krb5kdc[3435](info): set up 2 sockets
Sep 26 23:23:27 dist-test-slave-q5g8 krb5kdc[3435](info): commencing operation
krb5kdc: starting...
W20260926 23:23:28.385370 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:28 dist-test-slave-q5g8 krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:28.597260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34647
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:28.942261 3448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:28.942597 3448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:28.942771 3448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:28.950613 3448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:28.950743 3448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:28.950821 3448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:28.950933 3448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:28.951015 3448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:28.962113 3448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:28.963593 3448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:28.965111 3448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:28.978682 3453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:28.978660 3454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:28.979094 3456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:28.979601 3448 server_base.cc:1034] running on GCE node
I20260926 23:23:28.980541 3448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:28.982155 3448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:28.983395 3448 hybrid_clock.cc:648] HybridClock initialized: now 1790465008983347 us; error 80 us; skew 500 ppm
I20260926 23:23:28.983775 3448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:28 dist-test-slave-q5g8 krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:28.987073 3448 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:28.988624 3448 webserver.cc:460] Webserver started at http://127.0.97.62:34799/ using document root <none> and password file <none>
I20260926 23:23:28.989393 3448 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:28.989545 3448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:28.989876 3448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:28.993007 3448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "27e990b625e54fd2a10be9ef9d0e08e3"
format_stamp: "Formatted at 2026-09-26 23:23:28 on dist-test-slave-q5g8"
server_key: "ed5528168f8ac9e9de87e410c1e66f9c"
server_key_iv: "fc993b4d6250723704031555e830c41e"
server_key_version: "encryptionkey@0"
I20260926 23:23:28.993829 3448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "27e990b625e54fd2a10be9ef9d0e08e3"
format_stamp: "Formatted at 2026-09-26 23:23:28 on dist-test-slave-q5g8"
server_key: "ed5528168f8ac9e9de87e410c1e66f9c"
server_key_iv: "fc993b4d6250723704031555e830c41e"
server_key_version: "encryptionkey@0"
I20260926 23:23:28.999198 3448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 23:23:29.003113 3463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:29.004608 3448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:23:29.004943 3448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "27e990b625e54fd2a10be9ef9d0e08e3"
format_stamp: "Formatted at 2026-09-26 23:23:28 on dist-test-slave-q5g8"
server_key: "ed5528168f8ac9e9de87e410c1e66f9c"
server_key_iv: "fc993b4d6250723704031555e830c41e"
server_key_version: "encryptionkey@0"
I20260926 23:23:29.005184 3448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:29.024063 3448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:29.028808 3448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:29.029131 3448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:29.050803 3448 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34647
I20260926 23:23:29.050819 3514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34647 every 8 connection(s)
I20260926 23:23:29.053179 3448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:29.057430 3515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:29.058683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3448
I20260926 23:23:29.058948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:29.059677 388 external_mini_cluster.cc:1428] Setting key c77f023ca5a0e3c3f4adce3aebcc45b6
I20260926 23:23:29.069895 3515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3: Bootstrap starting.
I20260926 23:23:29.074019 3515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3: Neither blocks nor log segments found. Creating new log.
Sep 26 23:23:29 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:29.075279 3515 log.cc:824] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:29.079193 3515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3: No bootstrap required, opened a new log
I20260926 23:23:29.085723 3518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:29.061799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.0.97.62:34647)
0926 23:23:29.062726 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:29.062738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:29.062790 (+ 52us) server_negotiation.cc:407] Connection header received
0926 23:23:29.064528 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:29.064551 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:29.065058 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:29.065430 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:29.067213 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:29.068590 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:29.070380 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:29.071092 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:29.071232 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:29.076308 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:29.076318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:29.076336 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:29.076378 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:29.080460 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:29.081469 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:29.081479 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:29.081489 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:29.081726 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:29.082155 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:29.082162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:29.082169 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:29.082970 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:29.083275 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:29.083661 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 23:23:29.084151 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":435,"threads_started":1}
I20260926 23:23:29.089979 3515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260926 23:23:29.090376 3515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:29.090463 3515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e990b625e54fd2a10be9ef9d0e08e3, State: Initialized, Role: FOLLOWER
I20260926 23:23:29.090975 3515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260926 23:23:29.091171 3515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:29.091275 3515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:29.091405 3515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:29.094003 3515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260926 23:23:29.094534 3515 leader_election.cc:302] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e990b625e54fd2a10be9ef9d0e08e3; no voters:
I20260926 23:23:29.095105 3515 leader_election.cc:288] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:29.095352 3520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:29.096612 3520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [term 1 LEADER]: Becoming Leader. State: Replica: 27e990b625e54fd2a10be9ef9d0e08e3, State: Running, Role: LEADER
I20260926 23:23:29.097272 3520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260926 23:23:29.097671 3515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:29.100821 3521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } } }
I20260926 23:23:29.101204 3521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:29.100775 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e990b625e54fd2a10be9ef9d0e08e3. Latest consensus state: current_term: 1 leader_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e990b625e54fd2a10be9ef9d0e08e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } } }
I20260926 23:23:29.101976 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:29.112740 3526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:29.118441 3526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:29.129272 3526 catalog_manager.cc:1347] Generated new cluster ID: f09de12cccc84aefaf3b79859a0a8325
I20260926 23:23:29.129431 3526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:29.151054 3526 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:29.153072 3526 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:29.167964 3526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27e990b625e54fd2a10be9ef9d0e08e3: Generated new TSK 0
I20260926 23:23:29.169435 3526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:29.237967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:29.594605 3541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:29.594973 3541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:29.595144 3541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:29.603014 3541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:29.603188 3541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:29.603381 3541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:29.614972 3541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:29.616576 3541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:29.618104 3541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:29.632903 3549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:29.632875 3547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:29.632944 3546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:29.633129 3541 server_base.cc:1034] running on GCE node
I20260926 23:23:29.634253 3541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:29.635401 3541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:29.636667 3541 hybrid_clock.cc:648] HybridClock initialized: now 1790465009636616 us; error 51 us; skew 500 ppm
I20260926 23:23:29.637107 3541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:29 dist-test-slave-q5g8 krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:29.640944 3541 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:29.642622 3541 webserver.cc:460] Webserver started at http://127.0.97.1:37943/ using document root <none> and password file <none>
I20260926 23:23:29.643421 3541 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:29.643582 3541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:29.643920 3541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:29.647398 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "c6b4a89d867c483783eda4b94aedd301"
format_stamp: "Formatted at 2026-09-26 23:23:29 on dist-test-slave-q5g8"
server_key: "fb8e5f56a4a6974336404d9e39ba5e88"
server_key_iv: "98fe0db6c8083769f7c1d32bd9e9d615"
server_key_version: "encryptionkey@0"
I20260926 23:23:29.648300 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c6b4a89d867c483783eda4b94aedd301"
format_stamp: "Formatted at 2026-09-26 23:23:29 on dist-test-slave-q5g8"
server_key: "fb8e5f56a4a6974336404d9e39ba5e88"
server_key_iv: "98fe0db6c8083769f7c1d32bd9e9d615"
server_key_version: "encryptionkey@0"
I20260926 23:23:29.654096 3541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:23:29.657888 3556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:29.659482 3541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:29.659686 3541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "c6b4a89d867c483783eda4b94aedd301"
format_stamp: "Formatted at 2026-09-26 23:23:29 on dist-test-slave-q5g8"
server_key: "fb8e5f56a4a6974336404d9e39ba5e88"
server_key_iv: "98fe0db6c8083769f7c1d32bd9e9d615"
server_key_version: "encryptionkey@0"
I20260926 23:23:29.659910 3541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:29.694201 3541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:29.699069 3541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:29.699391 3541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:29.700462 3541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:29.701891 3541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:29.702004 3541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:29.702169 3541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:29.702241 3541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:29.736059 3541 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38637
I20260926 23:23:29.736141 3668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38637 every 8 connection(s)
I20260926 23:23:29.737499 3541 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:29.741626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3541
I20260926 23:23:29.741900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:29.742802 388 external_mini_cluster.cc:1428] Setting key d1a4757c8e8cbd691c6a67b4139074a2
Sep 26 23:23:29 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:29.761627 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:29.740257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55731 (local address 127.0.97.62:34647)
0926 23:23:29.740793 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:29.740797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:29.742041 (+ 1244us) server_negotiation.cc:407] Connection header received
0926 23:23:29.744526 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:29.744530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:29.744593 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:29.744755 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:29.746555 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:29.747761 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:29.749487 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:29.750179 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:29.750287 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:29.754489 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:29.754498 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:29.754506 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:29.754552 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:29.758202 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:29.759014 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:29.759020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:29.759024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:29.759179 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:29.759616 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:29.759623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:29.759629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:29.760039 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:29.760234 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:29.760971 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 23:23:29.761246 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:23:29.762701 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:29.740799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34647 (local address 127.0.97.1:55731)
0926 23:23:29.741714 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:29.741775 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:29.744207 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:29.745098 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:29.745112 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:29.745719 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:29.746360 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:29.746379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:29.747921 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:29.747925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:29.749330 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:29.749342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:29.750445 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:29.750455 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:29.750835 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:29.751700 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:29.751728 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:29.754219 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:29.758382 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:29.758389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:29.758405 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:29.758867 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:29.759302 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:29.759305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:29.759308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:29.759492 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:29.760181 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:29.760187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:29.760526 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 23:23:29.760851 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 23:23:29.761416 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":343,"threads_started":1}
I20260926 23:23:29.764323 3669 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34647
I20260926 23:23:29.764648 3669 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:29.765496 3669 heartbeater.cc:511] Master 127.0.97.62:34647 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:23:29.767920 3480 ts_manager.cc:194] Registered new tserver with Master: c6b4a89d867c483783eda4b94aedd301 (127.0.97.1:38637)
I20260926 23:23:29.769516 3480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:29.793394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:30.143685 3675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:30.144001 3675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:30.144110 3675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:30.151731 3675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:30.151895 3675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:30.152019 3675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:30.162981 3675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:30.164425 3675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:30.165910 3675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:30.180030 3680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:30.179999 3681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:30.180140 3683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:30.181012 3675 server_base.cc:1034] running on GCE node
I20260926 23:23:30.181769 3675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:30.182893 3675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:30.184137 3675 hybrid_clock.cc:648] HybridClock initialized: now 1790465010184062 us; error 68 us; skew 500 ppm
I20260926 23:23:30.184484 3675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:30.187870 3675 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:30.189489 3675 webserver.cc:460] Webserver started at http://127.0.97.2:35217/ using document root <none> and password file <none>
I20260926 23:23:30.190244 3675 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:30.190349 3675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:30.190596 3675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:30.193657 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "ed132266cb6d46b69c1ade4a22b75e72"
format_stamp: "Formatted at 2026-09-26 23:23:30 on dist-test-slave-q5g8"
server_key: "24bc6bef87c8b1f5e53cb3ca1aef1cfb"
server_key_iv: "f0df5b384d5d96e6cb355a1b5a3b0663"
server_key_version: "encryptionkey@0"
I20260926 23:23:30.194428 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ed132266cb6d46b69c1ade4a22b75e72"
format_stamp: "Formatted at 2026-09-26 23:23:30 on dist-test-slave-q5g8"
server_key: "24bc6bef87c8b1f5e53cb3ca1aef1cfb"
server_key_iv: "f0df5b384d5d96e6cb355a1b5a3b0663"
server_key_version: "encryptionkey@0"
I20260926 23:23:30.200069 3675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:23:30.204828 3690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:30.206379 3675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:30.206593 3675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "ed132266cb6d46b69c1ade4a22b75e72"
format_stamp: "Formatted at 2026-09-26 23:23:30 on dist-test-slave-q5g8"
server_key: "24bc6bef87c8b1f5e53cb3ca1aef1cfb"
server_key_iv: "f0df5b384d5d96e6cb355a1b5a3b0663"
server_key_version: "encryptionkey@0"
I20260926 23:23:30.206841 3675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:30.238207 3675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:30.242806 3675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:30.243119 3675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:30.244195 3675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:30.245718 3675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:30.245822 3675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:30.245976 3675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:30.246049 3675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:30.281348 3675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44375
I20260926 23:23:30.281409 3802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44375 every 8 connection(s)
I20260926 23:23:30.282771 3675 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:30.284500 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3675
I20260926 23:23:30.284747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:30.285315 388 external_mini_cluster.cc:1428] Setting key 0e9641c5ade29bdfcf1699e030c536d1
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:23:30.306547 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.286032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42715 (local address 127.0.97.62:34647)
0926 23:23:30.286496 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:30.286501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:30.287890 (+ 1389us) server_negotiation.cc:407] Connection header received
0926 23:23:30.289931 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:30.289937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:30.290050 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:30.290177 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:30.291859 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.292731 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.294425 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.295255 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.295408 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.299399 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:30.299408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:30.299417 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:30.299465 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:30.303304 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.304089 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.304096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.304100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.304267 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.304750 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.304755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.304758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.305136 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:30.305303 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:30.305966 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 23:23:30.306262 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":251,"threads_started":1}
I20260926 23:23:30.307691 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.286702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34647 (local address 127.0.97.2:42715)
0926 23:23:30.287728 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:30.287776 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:30.289637 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:30.290456 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:30.290465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:30.291047 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:30.291670 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:30.291687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.292903 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.292908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:30.294260 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:30.294273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.295481 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.295486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:30.295719 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.296512 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:30.296537 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:30.299122 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:30.303466 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:30.303475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:30.303491 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:30.303973 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:30.304395 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:30.304398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:30.304401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:30.304584 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:30.305267 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:30.305274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:30.305618 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 23:23:30.305870 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 23:23:30.306413 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":431,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:23:30.309422 3803 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34647
I20260926 23:23:30.309815 3803 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:30.310709 3803 heartbeater.cc:511] Master 127.0.97.62:34647 requested a full tablet report, sending...
I20260926 23:23:30.312419 3480 ts_manager.cc:194] Registered new tserver with Master: ed132266cb6d46b69c1ade4a22b75e72 (127.0.97.2:44375)
I20260926 23:23:30.313475 3480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:30.337073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:30.677107 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:30.677454 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:30.677635 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:30.685082 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:30.685266 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:30.685446 3809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:30.695654 3809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:30.697216 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:30.698763 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:30.713862 3815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:30.713958 3814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:30.713953 3817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:30.715302 3809 server_base.cc:1034] running on GCE node
I20260926 23:23:30.716025 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:30.717163 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:30.718423 3809 hybrid_clock.cc:648] HybridClock initialized: now 1790465010718361 us; error 53 us; skew 500 ppm
I20260926 23:23:30.718801 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:30.722450 3809 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:30.724119 3809 webserver.cc:460] Webserver started at http://127.0.97.3:37603/ using document root <none> and password file <none>
I20260926 23:23:30.724985 3809 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:30.725106 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:30.725389 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:30.728839 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "eb5c5d45fe2b4ba1844baec5e25afc6d"
format_stamp: "Formatted at 2026-09-26 23:23:30 on dist-test-slave-q5g8"
server_key: "048bfa3d2b1e0b3e9e87282135e87503"
server_key_iv: "7a4f58e46e1767dd90c595eeb784226a"
server_key_version: "encryptionkey@0"
I20260926 23:23:30.729750 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "eb5c5d45fe2b4ba1844baec5e25afc6d"
format_stamp: "Formatted at 2026-09-26 23:23:30 on dist-test-slave-q5g8"
server_key: "048bfa3d2b1e0b3e9e87282135e87503"
server_key_iv: "7a4f58e46e1767dd90c595eeb784226a"
server_key_version: "encryptionkey@0"
I20260926 23:23:30.735330 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 23:23:30.739300 3824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:30.740967 3809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 23:23:30.741168 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "eb5c5d45fe2b4ba1844baec5e25afc6d"
format_stamp: "Formatted at 2026-09-26 23:23:30 on dist-test-slave-q5g8"
server_key: "048bfa3d2b1e0b3e9e87282135e87503"
server_key_iv: "7a4f58e46e1767dd90c595eeb784226a"
server_key_version: "encryptionkey@0"
I20260926 23:23:30.741376 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:30.755923 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:30.760897 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:30.761252 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:30.762343 3809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:30.764029 3809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:30.764145 3809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:30.764307 3809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:30.764387 3809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:30.772799 3669 heartbeater.cc:503] Master 127.0.97.62:34647 was elected leader, sending a full tablet report...
I20260926 23:23:30.801750 3809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44629
I20260926 23:23:30.801789 3936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44629 every 8 connection(s)
I20260926 23:23:30.803095 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:30.807569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3809
I20260926 23:23:30.807791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:30.808338 388 external_mini_cluster.cc:1428] Setting key 2ea1d01701342114b4ad020b1fc25f29
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:30.825901 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.805920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38145 (local address 127.0.97.62:34647)
0926 23:23:30.806104 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:30.806109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:30.807429 (+ 1320us) server_negotiation.cc:407] Connection header received
0926 23:23:30.809789 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:30.809795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:30.809914 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:30.810022 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:30.812125 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.812980 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.814796 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.815272 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.815427 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.819532 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:30.819537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:30.819541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:30.819580 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:30.822362 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.823437 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.823441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.823443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.823618 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.824148 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.824153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.824159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.824568 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:30.824735 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:30.825471 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 23:23:30.825664 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:23:30.826992 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.806450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34647 (local address 127.0.97.3:38145)
0926 23:23:30.807229 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:30.807279 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:30.809474 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:30.810341 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:30.810355 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:30.811080 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:30.811944 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:30.811968 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.813154 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.813160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:30.814611 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:30.814625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.815553 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.815560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:30.815848 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.816602 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:30.816629 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:30.819263 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:30.822573 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:30.822583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:30.822604 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:30.823318 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:30.823769 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:30.823774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:30.823778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:30.824028 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:30.824733 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:30.824744 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:30.825132 (+ 388us) client_negotiation.cc:769] Sending connection context
0926 23:23:30.825380 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 23:23:30.825855 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":304,"threads_started":1}
I20260926 23:23:30.828514 3937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34647
I20260926 23:23:30.828895 3937 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:30.829695 3937 heartbeater.cc:511] Master 127.0.97.62:34647 requested a full tablet report, sending...
I20260926 23:23:30.831147 3480 ts_manager.cc:194] Registered new tserver with Master: eb5c5d45fe2b4ba1844baec5e25afc6d (127.0.97.3:44629)
I20260926 23:23:30.831961 3480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38145
I20260926 23:23:30.836932 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:30.885867 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.871837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.0.97.62:34647)
0926 23:23:30.872027 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:30.872032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:30.872531 (+ 499us) server_negotiation.cc:407] Connection header received
0926 23:23:30.872741 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:30.872745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:30.872812 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:30.872904 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:30.874357 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.875150 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.876399 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.876867 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.876965 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.880866 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:30.880872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:30.880875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:30.880913 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:30.883365 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.884007 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.884011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.884013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.884143 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.884537 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.884545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.884546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.884941 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:30.885129 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:30.885408 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 23:23:30.885596 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 23:23:30.886516 3480 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53434
I20260926 23:23:30.888338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3541
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:30.906656 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.889888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.0.97.1:38637)
0926 23:23:30.890477 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:30.890484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:30.890512 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:23:30.890575 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:30.890581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:30.890811 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:30.890946 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:30.892305 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.893631 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.895081 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.895748 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.895874 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.900333 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:30.900346 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:30.900362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:30.900403 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:30.903558 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.904362 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.904370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.904373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.904517 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.905082 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.905086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.905089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.905444 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:30.905692 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:30.905991 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 23:23:30.906271 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":286,"threads_started":1}
W20260926 23:23:30.908025 3664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42574
W20260926 23:23:30.909293 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:30.909510 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3541
I20260926 23:23:30.926267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3675
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:30.943738 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.927367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.0.97.2:44375)
0926 23:23:30.927952 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:30.927960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:30.927989 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:23:30.928092 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:30.928099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:30.928340 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:30.928482 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:30.929910 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.930996 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.932335 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.933119 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.933231 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.937598 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:30.937613 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:30.937630 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:30.937672 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:30.940653 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.941406 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.941413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.941416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.941565 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.942030 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.942034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.942036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.942397 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:30.942656 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:30.943019 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 23:23:30.943350 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":312,"threads_started":1}
W20260926 23:23:30.945420 3798 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53008
W20260926 23:23:30.946738 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3675: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:30.946919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3675
I20260926 23:23:30.963564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3809
Sep 26 23:23:30 dist-test-slave-q5g8 krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:30.983162 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:30.964867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.0.97.3:44629)
0926 23:23:30.965729 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:30.965736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:30.965770 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:23:30.965838 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:30.965844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:30.966148 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:30.966319 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:30.967875 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.969241 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.970796 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:30.971462 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:30.971591 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:30.976910 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:30.976926 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:30.976943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:30.977030 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:30.980091 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.980955 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.980964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.980966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.981121 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:30.981529 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:30.981532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:30.981534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:30.981907 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:30.982178 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:30.982457 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 23:23:30.982741 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":504,"threads_started":1}
W20260926 23:23:30.984683 3932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58430
W20260926 23:23:30.986303 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3809: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:23:30.986460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3809
I20260926 23:23:31.002732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3448
I20260926 23:23:31.003962 3510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:31.109493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3448
2026-09-26T23:23:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3868 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:31 dist-test-slave-q5g8 krb5kdc[3954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:31 dist-test-slave-q5g8 krb5kdc[3954](info): set up 2 sockets
Sep 26 23:23:31 dist-test-slave-q5g8 krb5kdc[3954](info): commencing operation
krb5kdc: starting...
W20260926 23:23:32.245805 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:32 dist-test-slave-q5g8 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:32.464286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37103
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:32.804033 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:32.804361 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:32.804528 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:32.812333 3967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:32.812462 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:32.812587 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:32.812675 3967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:32.812783 3967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:32.822984 3967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:32.824374 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:32.825743 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:32.837690 3972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:32.837666 3973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:32.837877 3975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:32.838536 3967 server_base.cc:1034] running on GCE node
I20260926 23:23:32.839327 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:32.840804 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:32.842038 3967 hybrid_clock.cc:648] HybridClock initialized: now 1790465012841977 us; error 67 us; skew 500 ppm
I20260926 23:23:32.842399 3967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:32 dist-test-slave-q5g8 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:32.850049 3967 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:32.851752 3967 webserver.cc:460] Webserver started at http://127.0.97.62:38311/ using document root <none> and password file <none>
I20260926 23:23:32.852557 3967 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:32.852664 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:32.853066 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:32.856168 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "fa1b2ce404c442f7ab21a892ae4591cf"
format_stamp: "Formatted at 2026-09-26 23:23:32 on dist-test-slave-q5g8"
server_key: "079229ecc1ec7498ad5701aa05ab96ad"
server_key_iv: "24fda29fccc4c0aa3c18a521e59c2882"
server_key_version: "encryptionkey@0"
I20260926 23:23:32.857035 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "fa1b2ce404c442f7ab21a892ae4591cf"
format_stamp: "Formatted at 2026-09-26 23:23:32 on dist-test-slave-q5g8"
server_key: "079229ecc1ec7498ad5701aa05ab96ad"
server_key_iv: "24fda29fccc4c0aa3c18a521e59c2882"
server_key_version: "encryptionkey@0"
I20260926 23:23:32.862249 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 23:23:32.865809 3982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:32.867201 3967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:23:32.867388 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "fa1b2ce404c442f7ab21a892ae4591cf"
format_stamp: "Formatted at 2026-09-26 23:23:32 on dist-test-slave-q5g8"
server_key: "079229ecc1ec7498ad5701aa05ab96ad"
server_key_iv: "24fda29fccc4c0aa3c18a521e59c2882"
server_key_version: "encryptionkey@0"
I20260926 23:23:32.867591 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:32.893736 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:32.898129 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:32.898432 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:32.917235 3967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37103
I20260926 23:23:32.917268 4033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37103 every 8 connection(s)
I20260926 23:23:32.918624 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:32.922703 4034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:32.924539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 3967
I20260926 23:23:32.924782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:32.925431 388 external_mini_cluster.cc:1428] Setting key 2db803c6ebc65eb2877d2b802f81bc87
I20260926 23:23:32.935794 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf: Bootstrap starting.
I20260926 23:23:32.940016 4034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf: Neither blocks nor log segments found. Creating new log.
Sep 26 23:23:32 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:32.941289 4034 log.cc:824] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:32.945009 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf: No bootstrap required, opened a new log
I20260926 23:23:32.950111 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:32.927747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.0.97.62:37103)
0926 23:23:32.928635 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:32.928648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:32.928766 (+ 118us) server_negotiation.cc:407] Connection header received
0926 23:23:32.930532 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:32.930558 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:32.930980 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:32.931343 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:32.933199 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:32.934596 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:32.936344 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:32.937080 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:32.937226 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:32.942090 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:32.942107 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:32.942138 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:32.942199 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:32.945844 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:32.946597 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:32.946604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:32.946614 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:32.946789 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:32.947194 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:32.947199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:32.947204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:32.947760 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:32.948079 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:32.948547 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 23:23:32.949030 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":378,"threads_started":1}
I20260926 23:23:32.953231 4034 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37103 } }
I20260926 23:23:32.953671 4034 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:32.953792 4034 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1b2ce404c442f7ab21a892ae4591cf, State: Initialized, Role: FOLLOWER
I20260926 23:23:32.954434 4034 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37103 } }
I20260926 23:23:32.954694 4034 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:32.954856 4034 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:32.955052 4034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:32.958307 4034 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37103 } }
I20260926 23:23:32.958941 4034 leader_election.cc:302] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa1b2ce404c442f7ab21a892ae4591cf; no voters:
I20260926 23:23:32.959592 4034 leader_election.cc:288] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:32.959755 4039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:32.960606 4039 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [term 1 LEADER]: Becoming Leader. State: Replica: fa1b2ce404c442f7ab21a892ae4591cf, State: Running, Role: LEADER
I20260926 23:23:32.961365 4039 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37103 } }
I20260926 23:23:32.962081 4034 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:32.964664 4041 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa1b2ce404c442f7ab21a892ae4591cf. Latest consensus state: current_term: 1 leader_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37103 } } }
I20260926 23:23:32.964664 4040 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b2ce404c442f7ab21a892ae4591cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37103 } } }
I20260926 23:23:32.965183 4040 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:32.965183 4041 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:32.966048 4044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:32.971906 4044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:32.982895 4044 catalog_manager.cc:1347] Generated new cluster ID: 41e63422830f4567a5362e2fdfcb942e
I20260926 23:23:32.983023 4044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:33.024871 4044 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:33.026338 4044 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:33.037575 4044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa1b2ce404c442f7ab21a892ae4591cf: Generated new TSK 0
I20260926 23:23:33.038527 4044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:33.097566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37103
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:33.444056 4060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:33.444370 4060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:33.444465 4060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:33.452356 4060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:33.452530 4060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:33.452651 4060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:33.463292 4060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:33.464910 4060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:33.466317 4060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:33.480762 4066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:33.480747 4068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:33.480782 4065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:33.482100 4060 server_base.cc:1034] running on GCE node
I20260926 23:23:33.482793 4060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:33.483821 4060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:33.485044 4060 hybrid_clock.cc:648] HybridClock initialized: now 1790465013484981 us; error 62 us; skew 500 ppm
I20260926 23:23:33.485363 4060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:33 dist-test-slave-q5g8 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:33.488629 4060 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:33.490302 4060 webserver.cc:460] Webserver started at http://127.0.97.1:39479/ using document root <none> and password file <none>
I20260926 23:23:33.490999 4060 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:33.491104 4060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:33.491365 4060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:33.494362 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "6b828ea3e57541c3bddaa524d8c493da"
format_stamp: "Formatted at 2026-09-26 23:23:33 on dist-test-slave-q5g8"
server_key: "239ad83e37b2586db0268d0185d4cece"
server_key_iv: "fdea14ac4a3b0ac1a0dbe7191bf9dc2e"
server_key_version: "encryptionkey@0"
I20260926 23:23:33.495113 4060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6b828ea3e57541c3bddaa524d8c493da"
format_stamp: "Formatted at 2026-09-26 23:23:33 on dist-test-slave-q5g8"
server_key: "239ad83e37b2586db0268d0185d4cece"
server_key_iv: "fdea14ac4a3b0ac1a0dbe7191bf9dc2e"
server_key_version: "encryptionkey@0"
I20260926 23:23:33.500360 4060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:23:33.503958 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:33.505520 4060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:33.505657 4060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "6b828ea3e57541c3bddaa524d8c493da"
format_stamp: "Formatted at 2026-09-26 23:23:33 on dist-test-slave-q5g8"
server_key: "239ad83e37b2586db0268d0185d4cece"
server_key_iv: "fdea14ac4a3b0ac1a0dbe7191bf9dc2e"
server_key_version: "encryptionkey@0"
I20260926 23:23:33.505828 4060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:33.531946 4060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:33.536060 4060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:33.536304 4060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:33.537405 4060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:33.538714 4060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:33.538785 4060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:33.538926 4060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:33.538987 4060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:33.573148 4060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44437
I20260926 23:23:33.573220 4187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44437 every 8 connection(s)
I20260926 23:23:33.574419 4060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:33.581264 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 4060
I20260926 23:23:33.581480 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:33.582123 388 external_mini_cluster.cc:1428] Setting key 09b0f2141d9872479a0ca72baffee4e4
Sep 26 23:23:33 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:33.594688 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:33.576981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32967 (local address 127.0.97.62:37103)
0926 23:23:33.577463 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:33.577467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:33.578363 (+ 896us) server_negotiation.cc:407] Connection header received
0926 23:23:33.580176 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:33.580181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:33.580281 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:33.580399 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:33.582146 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:33.583052 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:33.584834 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:33.585252 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:33.585388 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:33.588957 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:33.588961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:33.588966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:33.588997 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:33.591588 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:33.592344 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:33.592354 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:33.592356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:33.592469 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:33.592972 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:33.592977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:33.592982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:33.593241 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:33.593390 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:33.594111 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 23:23:33.594336 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
I20260926 23:23:33.595710 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:33.577447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37103 (local address 127.0.97.1:32967)
0926 23:23:33.578204 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:33.578245 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:33.579902 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:33.580650 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:33.580660 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:33.581279 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:33.581975 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:33.581993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:33.583221 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:33.583225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:33.584626 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:33.584639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:33.585500 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:33.585507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:33.585755 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:33.586469 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:33.586493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:33.588649 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:33.591749 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:33.591759 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:33.591779 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:33.592211 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:33.592580 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:33.592585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:33.592588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:33.592854 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:33.593373 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:33.593380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:33.593740 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 23:23:33.594007 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 23:23:33.594558 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":314,"threads_started":1}
I20260926 23:23:33.597442 4188 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37103
I20260926 23:23:33.597801 4188 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:33.598721 4188 heartbeater.cc:511] Master 127.0.97.62:37103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:23:33.601069 3999 ts_manager.cc:194] Registered new tserver with Master: 6b828ea3e57541c3bddaa524d8c493da (127.0.97.1:44437)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:23:33.602519 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:33.627743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37103
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:33.960798 4194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:33.961127 4194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:33.961323 4194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:33.969710 4194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:33.969853 4194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:33.970028 4194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:33.980566 4194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:33.982013 4194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:33.983384 4194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:33.997244 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:33.997267 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:33.997319 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:33.998584 4194 server_base.cc:1034] running on GCE node
I20260926 23:23:33.999317 4194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:34.000485 4194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:34.001734 4194 hybrid_clock.cc:648] HybridClock initialized: now 1790465014001645 us; error 87 us; skew 500 ppm
I20260926 23:23:34.002094 4194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:34.005784 4194 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:34.007601 4194 webserver.cc:460] Webserver started at http://127.0.97.2:42131/ using document root <none> and password file <none>
I20260926 23:23:34.008361 4194 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:34.008476 4194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:34.008852 4194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:34.012167 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "9cf8acae06ff47889e0b49c2da3afcae"
format_stamp: "Formatted at 2026-09-26 23:23:34 on dist-test-slave-q5g8"
server_key: "68bf1dce060721201158f959bca155ce"
server_key_iv: "560a66bc7a9c3e4d37473adb8458997d"
server_key_version: "encryptionkey@0"
I20260926 23:23:34.013087 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9cf8acae06ff47889e0b49c2da3afcae"
format_stamp: "Formatted at 2026-09-26 23:23:34 on dist-test-slave-q5g8"
server_key: "68bf1dce060721201158f959bca155ce"
server_key_iv: "560a66bc7a9c3e4d37473adb8458997d"
server_key_version: "encryptionkey@0"
I20260926 23:23:34.018865 4194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:34.022615 4209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:34.024144 4194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:23:34.024302 4194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "9cf8acae06ff47889e0b49c2da3afcae"
format_stamp: "Formatted at 2026-09-26 23:23:34 on dist-test-slave-q5g8"
server_key: "68bf1dce060721201158f959bca155ce"
server_key_iv: "560a66bc7a9c3e4d37473adb8458997d"
server_key_version: "encryptionkey@0"
I20260926 23:23:34.024483 4194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:34.041714 4194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:34.046727 4194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:34.047051 4194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:34.048177 4194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:34.049809 4194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:34.049898 4194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:34.050005 4194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:34.050056 4194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:34.087919 4194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43587
I20260926 23:23:34.087937 4321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43587 every 8 connection(s)
I20260926 23:23:34.089671 4194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:34.098091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 4194
I20260926 23:23:34.098325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:34.098839 388 external_mini_cluster.cc:1428] Setting key 429537e42c2d0b0a3b72d373968b7fe4
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:34.114717 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.093310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58563 (local address 127.0.97.62:37103)
0926 23:23:34.093512 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:34.093517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:34.095054 (+ 1537us) server_negotiation.cc:407] Connection header received
0926 23:23:34.097518 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:34.097524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:34.097640 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:34.097768 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:34.100065 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.101083 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.103034 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.103879 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.104045 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.108042 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:34.108048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:34.108052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:34.108088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:34.111186 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.112048 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.112057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.112061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.112211 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.112764 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.112772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.112775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.113069 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:34.113241 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:34.114077 (+ 836us) server_negotiation.cc:299] Negotiation successful
0926 23:23:34.114347 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:23:34.116077 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.093855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37103 (local address 127.0.97.2:58563)
0926 23:23:34.094836 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:34.094894 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:34.097142 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:34.098072 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:34.098083 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:34.098892 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:34.099861 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:34.099886 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.101297 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.101305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:34.102840 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:34.102855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.104177 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.104183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:34.104421 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.105321 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:34.105348 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:34.107747 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:34.111367 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:34.111379 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:34.111400 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:34.111897 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:34.112368 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:34.112373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:34.112377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:34.112605 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:34.113220 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:34.113229 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:34.113649 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 23:23:34.113972 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 23:23:34.114602 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":378,"threads_started":1}
I20260926 23:23:34.117868 4322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37103
I20260926 23:23:34.118274 4322 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:23:34.119405 4322 heartbeater.cc:511] Master 127.0.97.62:37103 requested a full tablet report, sending...
I20260926 23:23:34.121271 3999 ts_manager.cc:194] Registered new tserver with Master: 9cf8acae06ff47889e0b49c2da3afcae (127.0.97.2:43587)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:23:34.122716 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:34.148782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37103
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:34.506168 4327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:34.506500 4327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:34.506597 4327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:34.514408 4327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:34.514544 4327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:34.514662 4327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:34.524998 4327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:34.526408 4327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:34.527808 4327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:34.542253 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:34.542253 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:34.543118 4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:34.543989 4327 server_base.cc:1034] running on GCE node
I20260926 23:23:34.544840 4327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:34.545912 4327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:34.547129 4327 hybrid_clock.cc:648] HybridClock initialized: now 1790465014547078 us; error 38 us; skew 500 ppm
I20260926 23:23:34.547528 4327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:34.550962 4327 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:34.552534 4327 webserver.cc:460] Webserver started at http://127.0.97.3:39715/ using document root <none> and password file <none>
I20260926 23:23:34.553313 4327 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:34.553452 4327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:34.553792 4327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:34.556912 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "4ae22b0d3d4542058b97897dbb230d70"
format_stamp: "Formatted at 2026-09-26 23:23:34 on dist-test-slave-q5g8"
server_key: "1f7a9f2b4b693bec7e025baf15a5852e"
server_key_iv: "81c5d200a496e5d24703a531fb8065b5"
server_key_version: "encryptionkey@0"
I20260926 23:23:34.557766 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4ae22b0d3d4542058b97897dbb230d70"
format_stamp: "Formatted at 2026-09-26 23:23:34 on dist-test-slave-q5g8"
server_key: "1f7a9f2b4b693bec7e025baf15a5852e"
server_key_iv: "81c5d200a496e5d24703a531fb8065b5"
server_key_version: "encryptionkey@0"
I20260926 23:23:34.563640 4327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:23:34.567557 4342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:34.569495 4327 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:23:34.569749 4327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "4ae22b0d3d4542058b97897dbb230d70"
format_stamp: "Formatted at 2026-09-26 23:23:34 on dist-test-slave-q5g8"
server_key: "1f7a9f2b4b693bec7e025baf15a5852e"
server_key_iv: "81c5d200a496e5d24703a531fb8065b5"
server_key_version: "encryptionkey@0"
I20260926 23:23:34.570003 4327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:34.584988 4327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:34.589605 4327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:34.589926 4327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:34.590967 4327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:34.592422 4327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:34.592495 4327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:34.592643 4327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:34.592739 4327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 23:23:34.605592 4188 heartbeater.cc:503] Master 127.0.97.62:37103 was elected leader, sending a full tablet report...
I20260926 23:23:34.627911 4327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34207
I20260926 23:23:34.628006 4454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34207 every 8 connection(s)
I20260926 23:23:34.629475 4327 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:34.631742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 4327
I20260926 23:23:34.631916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:34.632395 388 external_mini_cluster.cc:1428] Setting key 3550b501614311c6542871853f8faf04
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:34.653375 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.632560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42751 (local address 127.0.97.62:37103)
0926 23:23:34.633093 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:34.633101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:34.634570 (+ 1469us) server_negotiation.cc:407] Connection header received
0926 23:23:34.637119 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:34.637123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:34.637192 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:34.637286 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:34.638986 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.639750 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.641591 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.641956 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.642050 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.646013 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:34.646019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:34.646022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:34.646058 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:34.649444 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.650400 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.650405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.650407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.650552 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.651041 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.651045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.651047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.651358 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:34.651486 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:34.652382 (+ 896us) server_negotiation.cc:299] Negotiation successful
0926 23:23:34.652778 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":260,"threads_started":1}
I20260926 23:23:34.654458 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.633375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37103 (local address 127.0.97.3:42751)
0926 23:23:34.634408 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:34.634460 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:34.636839 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:34.637546 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:34.637555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:34.638181 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:34.638827 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:34.638844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.639902 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.639906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:34.641436 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:34.641453 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.642181 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.642187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:34.642411 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.643221 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:34.643246 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:34.645730 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:34.649658 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:34.649669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:34.649686 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:34.650232 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:34.650663 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:34.650667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:34.650669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:34.650908 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:34.651477 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:34.651485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:34.651899 (+ 414us) client_negotiation.cc:769] Sending connection context
0926 23:23:34.652175 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 23:23:34.652920 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":455,"threads_started":1}
I20260926 23:23:34.656222 4455 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37103
I20260926 23:23:34.656605 4455 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:34.657482 4455 heartbeater.cc:511] Master 127.0.97.62:37103 requested a full tablet report, sending...
I20260926 23:23:34.659368 3999 ts_manager.cc:194] Registered new tserver with Master: 4ae22b0d3d4542058b97897dbb230d70 (127.0.97.3:34207)
I20260926 23:23:34.660457 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42751
I20260926 23:23:34.661193 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:34.715190 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.697982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.97.1:44437)
0926 23:23:34.698669 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:34.698677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:34.698705 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:23:34.698772 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:34.698779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:34.699006 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:34.699201 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:34.700509 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.701788 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.703110 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.703767 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.703881 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.708165 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:34.708182 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:34.708200 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:34.708258 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:34.711911 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.712771 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.712779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.712782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.712933 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.713392 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.713398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.713403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.713810 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:34.714162 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:34.714508 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 23:23:34.714803 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":355,"threads_started":1}
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:34.735092 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.720191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.0.97.62:37103)
0926 23:23:34.720416 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:34.720423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:34.720609 (+ 186us) server_negotiation.cc:407] Connection header received
0926 23:23:34.720863 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:34.720869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:34.720948 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:34.721053 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:34.722292 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.723031 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.724327 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.724837 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.724977 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.729225 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:34.729232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:34.729236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:34.729278 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:34.732026 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.732911 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.732918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.732922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.733066 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.733513 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.733520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.733529 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.733878 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:34.734154 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:34.734510 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 23:23:34.734757 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 23:23:34.735875 3999 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43518
I20260926 23:23:34.737207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 4060
I20260926 23:23:34.751582 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.738233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.97.1:44437)
0926 23:23:34.738418 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:34.738423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:34.738754 (+ 331us) server_negotiation.cc:407] Connection header received
0926 23:23:34.738907 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:34.738911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:34.739024 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:34.739124 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:34.740495 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.741438 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.742786 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.743230 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.743339 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.746082 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:34.746088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:34.746092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:34.746130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:34.748961 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.749669 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.749673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.749675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.749843 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.750261 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.750266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.750268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.750550 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:34.750812 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:34.751161 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 23:23:34.751344 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:23:34.752290 4183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:34.841733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 4060
I20260926 23:23:34.859886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 4194
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:34.876874 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.861039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.0.97.2:43587)
0926 23:23:34.861634 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:34.861642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:34.861668 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:23:34.861732 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:34.861741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:34.861971 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:34.862161 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:34.863470 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.864565 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.865927 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.866489 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.866637 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.871037 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:34.871064 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:34.871082 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:34.871133 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:34.874078 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.874778 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.874786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.874789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.874911 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.875217 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.875220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.875222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.875658 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:34.875888 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:34.876243 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 23:23:34.876487 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":323,"threads_started":1}
I20260926 23:23:34.878151 4317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:34.963254 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 4194
I20260926 23:23:34.980069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 4327
Sep 26 23:23:34 dist-test-slave-q5g8 krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:34.998885 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:34.981283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.0.97.3:34207)
0926 23:23:34.982077 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:34.982089 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:34.982140 (+ 51us) server_negotiation.cc:407] Connection header received
0926 23:23:34.982207 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:34.982217 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:34.982458 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:34.982667 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:34.984134 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.985504 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.987007 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:34.987677 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:34.987853 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:34.992772 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:34.992787 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:34.992804 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:34.992845 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:34.995901 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.996675 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.996742 (+ 67us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.996745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.996874 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:34.997324 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:34.997327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:34.997329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:34.997658 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:34.997925 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:34.998254 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 23:23:34.998456 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":343,"threads_started":1}
I20260926 23:23:35.000332 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:35.084775 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 4327
I20260926 23:23:35.101629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 3967
I20260926 23:23:35.103134 4029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:35.209522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 3967
2026-09-26T23:23:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4102 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:35 dist-test-slave-q5g8 krb5kdc[4475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:35 dist-test-slave-q5g8 krb5kdc[4475](info): set up 2 sockets
Sep 26 23:23:35 dist-test-slave-q5g8 krb5kdc[4475](info): commencing operation
krb5kdc: starting...
W20260926 23:23:36.323025 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:36 dist-test-slave-q5g8 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:36.540369 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42255
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:36.891068 4488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:36.891495 4488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:36.891691 4488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:36.899801 4488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:36.899969 4488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:36.900084 4488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:36.900187 4488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:36.900283 4488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:36.910921 4488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:36.912398 4488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:36.913947 4488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:36.926890 4496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:36.926841 4493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:36.926815 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:36.927788 4488 server_base.cc:1034] running on GCE node
I20260926 23:23:36.928660 4488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:36.930444 4488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:36.931713 4488 hybrid_clock.cc:648] HybridClock initialized: now 1790465016931681 us; error 65 us; skew 500 ppm
I20260926 23:23:36.932133 4488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:36 dist-test-slave-q5g8 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:36.935460 4488 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:36.936868 4488 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:36.936965 4488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:36.937247 4488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:36.940433 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "2464649de6c744c79c003a82337632c9"
format_stamp: "Formatted at 2026-09-26 23:23:36 on dist-test-slave-q5g8"
server_key: "ab12901ca97b4b01c45e95dd4ade810f"
server_key_iv: "1dbe2890e10943a40aed84fbc5dd4eab"
server_key_version: "encryptionkey@0"
I20260926 23:23:36.941329 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "2464649de6c744c79c003a82337632c9"
format_stamp: "Formatted at 2026-09-26 23:23:36 on dist-test-slave-q5g8"
server_key: "ab12901ca97b4b01c45e95dd4ade810f"
server_key_iv: "1dbe2890e10943a40aed84fbc5dd4eab"
server_key_version: "encryptionkey@0"
I20260926 23:23:36.947067 4488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:36.950692 4502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:36.952229 4488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:23:36.952416 4488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "2464649de6c744c79c003a82337632c9"
format_stamp: "Formatted at 2026-09-26 23:23:36 on dist-test-slave-q5g8"
server_key: "ab12901ca97b4b01c45e95dd4ade810f"
server_key_iv: "1dbe2890e10943a40aed84fbc5dd4eab"
server_key_version: "encryptionkey@0"
I20260926 23:23:36.952664 4488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:36.973627 4488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:36.977866 4488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:36.978173 4488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:36.997236 4488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42255
I20260926 23:23:36.997257 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42255 every 8 connection(s)
I20260926 23:23:36.998167 4488 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:37.001853 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 4488
I20260926 23:23:37.002068 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:37.002183 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:37.002923 388 external_mini_cluster.cc:1428] Setting key 8138ba368351612bee74bff760f4ab25
I20260926 23:23:37.017812 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9: Bootstrap starting.
Sep 26 23:23:37 dist-test-slave-q5g8 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:37.021593 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:37.022729 4554 log.cc:824] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:37.025872 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9: No bootstrap required, opened a new log
I20260926 23:23:37.026937 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:37.005084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.0.97.62:42255)
0926 23:23:37.005955 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:37.005966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:37.006010 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:23:37.007505 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:37.007535 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:37.007979 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:37.008353 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:37.010323 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:37.011667 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:37.013559 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:37.014321 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:37.014472 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:37.019263 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:37.019272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:37.019294 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:37.019348 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:37.022586 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:37.023388 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:37.023395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:37.023401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:37.023595 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:37.024010 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:37.024023 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:37.024025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:37.024567 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:37.024917 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:37.025359 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 23:23:37.025796 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":422,"threads_started":1}
I20260926 23:23:37.036070 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464649de6c744c79c003a82337632c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260926 23:23:37.036690 4554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:37.036872 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2464649de6c744c79c003a82337632c9, State: Initialized, Role: FOLLOWER
I20260926 23:23:37.037813 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464649de6c744c79c003a82337632c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260926 23:23:37.038218 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:37.038420 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:37.038652 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:37.041847 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464649de6c744c79c003a82337632c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260926 23:23:37.042438 4554 leader_election.cc:302] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2464649de6c744c79c003a82337632c9; no voters:
I20260926 23:23:37.043149 4554 leader_election.cc:288] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:37.043268 4559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:37.044080 4559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2464649de6c744c79c003a82337632c9, State: Running, Role: LEADER
I20260926 23:23:37.044628 4559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464649de6c744c79c003a82337632c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260926 23:23:37.045660 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:37.047816 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2464649de6c744c79c003a82337632c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464649de6c744c79c003a82337632c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } } }
I20260926 23:23:37.048305 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:37.049263 4561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2464649de6c744c79c003a82337632c9. Latest consensus state: current_term: 1 leader_uuid: "2464649de6c744c79c003a82337632c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464649de6c744c79c003a82337632c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } } }
I20260926 23:23:37.049728 4561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:37.050482 4564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:37.056593 4564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:37.066860 4564 catalog_manager.cc:1347] Generated new cluster ID: 3d86c63f46f646878030dbb6c5b09d15
I20260926 23:23:37.066992 4564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:37.098582 4564 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:37.100355 4564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:37.112844 4564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9: Generated new TSK 0
I20260926 23:23:37.114090 4564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:37.177167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:37.535454 4580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:37.535827 4580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:37.536006 4580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:37.543623 4580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:37.543816 4580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:37.544023 4580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:37.554908 4580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:37.556535 4580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:37.558091 4580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:37.573158 4585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:37.573200 4586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:37.573148 4588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:37.574827 4580 server_base.cc:1034] running on GCE node
I20260926 23:23:37.575634 4580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:37.576879 4580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:37.578150 4580 hybrid_clock.cc:648] HybridClock initialized: now 1790465017578095 us; error 56 us; skew 500 ppm
I20260926 23:23:37.578603 4580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:37 dist-test-slave-q5g8 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:37.582388 4580 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:37.583760 4580 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:37.583869 4580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:37.584168 4580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:37.587872 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "cf40c915cd2d44ba81a6065b0c9b9384"
format_stamp: "Formatted at 2026-09-26 23:23:37 on dist-test-slave-q5g8"
server_key: "83ab3f1f28d51a0aa468ab45669b43ed"
server_key_iv: "fbe7a99031c57c2e64ecae971b575707"
server_key_version: "encryptionkey@0"
I20260926 23:23:37.588862 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cf40c915cd2d44ba81a6065b0c9b9384"
format_stamp: "Formatted at 2026-09-26 23:23:37 on dist-test-slave-q5g8"
server_key: "83ab3f1f28d51a0aa468ab45669b43ed"
server_key_iv: "fbe7a99031c57c2e64ecae971b575707"
server_key_version: "encryptionkey@0"
I20260926 23:23:37.594499 4580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:23:37.598376 4594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:37.600085 4580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:23:37.600322 4580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "cf40c915cd2d44ba81a6065b0c9b9384"
format_stamp: "Formatted at 2026-09-26 23:23:37 on dist-test-slave-q5g8"
server_key: "83ab3f1f28d51a0aa468ab45669b43ed"
server_key_iv: "fbe7a99031c57c2e64ecae971b575707"
server_key_version: "encryptionkey@0"
I20260926 23:23:37.600598 4580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:37.635426 4580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:37.640350 4580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:37.640728 4580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:37.641850 4580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:37.643318 4580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:37.643435 4580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:37.643575 4580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:37.643680 4580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:37.679975 4580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45917
I20260926 23:23:37.680033 4706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45917 every 8 connection(s)
I20260926 23:23:37.681021 4580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:37.681862 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 4580
I20260926 23:23:37.682118 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:37.682893 388 external_mini_cluster.cc:1428] Setting key a981153502ff30208e42816f4cb169c7
Sep 26 23:23:37 dist-test-slave-q5g8 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:23:37.709779 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:37.684056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49911 (local address 127.0.97.62:42255)
0926 23:23:37.684548 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:37.684555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:37.685895 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 23:23:37.687911 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:37.687916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:37.688008 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:37.688145 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:37.690132 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:37.690938 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:37.692867 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:37.693335 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:37.693495 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:37.702595 (+ 9100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:37.702601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:37.702605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:37.702646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:37.705799 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:37.706928 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:37.706935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:37.706941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:37.707094 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:37.707552 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:37.707560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:37.707566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:37.707992 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:37.708169 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:37.709011 (+ 842us) server_negotiation.cc:299] Negotiation successful
0926 23:23:37.709273 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260926 23:23:37.710481 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:37.684845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.1:49911)
0926 23:23:37.685715 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:37.685764 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:37.687585 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:37.688455 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:37.688466 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:37.689203 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:37.689885 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:37.689907 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:37.691151 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:37.691155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:37.692601 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:37.692617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:37.693638 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:37.693646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:37.693915 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:37.699025 (+ 5110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:37.699057 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:37.702216 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:37.706014 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:37.706023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:37.706078 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:37.706748 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:37.707214 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:37.707218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:37.707220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:37.707420 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:37.708132 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:37.708139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:37.708511 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 23:23:37.708848 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 23:23:37.709297 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":381,"threads_started":1}
I20260926 23:23:37.712236 4707 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260926 23:23:37.712613 4707 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:37.713543 4707 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260926 23:23:37.715695 4519 ts_manager.cc:194] Registered new tserver with Master: cf40c915cd2d44ba81a6065b0c9b9384 (127.0.97.1:45917)
I20260926 23:23:37.717212 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:37.732754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:38.086717 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:38.087064 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:38.087174 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:38.095422 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:38.095569 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:38.095691 4713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:38.106731 4713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:38.108201 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:38.109932 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:38.123725 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:38.123770 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:38.123934 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:38.125151 4713 server_base.cc:1034] running on GCE node
I20260926 23:23:38.125859 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:38.126929 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:38.128207 4713 hybrid_clock.cc:648] HybridClock initialized: now 1790465018128144 us; error 51 us; skew 500 ppm
I20260926 23:23:38.128633 4713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:38 dist-test-slave-q5g8 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:38.142202 4713 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:38.143625 4713 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:38.143729 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:38.144032 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:38.147492 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "9e9f8f16db34428db5a11f9016630bbf"
format_stamp: "Formatted at 2026-09-26 23:23:38 on dist-test-slave-q5g8"
server_key: "37e7fe114b26f98622a9ee48c807651f"
server_key_iv: "5cf50e97ade92a34a6de09d614635bd8"
server_key_version: "encryptionkey@0"
I20260926 23:23:38.148339 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9e9f8f16db34428db5a11f9016630bbf"
format_stamp: "Formatted at 2026-09-26 23:23:38 on dist-test-slave-q5g8"
server_key: "37e7fe114b26f98622a9ee48c807651f"
server_key_iv: "5cf50e97ade92a34a6de09d614635bd8"
server_key_version: "encryptionkey@0"
I20260926 23:23:38.154173 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 23:23:38.158190 4727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:38.159659 4713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:38.159868 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "9e9f8f16db34428db5a11f9016630bbf"
format_stamp: "Formatted at 2026-09-26 23:23:38 on dist-test-slave-q5g8"
server_key: "37e7fe114b26f98622a9ee48c807651f"
server_key_iv: "5cf50e97ade92a34a6de09d614635bd8"
server_key_version: "encryptionkey@0"
I20260926 23:23:38.160115 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:38.179811 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:38.184929 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:38.185248 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:38.186334 4713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:38.187850 4713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:38.187976 4713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:38.188133 4713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:38.188241 4713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:38.225525 4713 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39119
I20260926 23:23:38.225622 4839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39119 every 8 connection(s)
I20260926 23:23:38.226480 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:38.234782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 4713
I20260926 23:23:38.234983 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:38.235632 388 external_mini_cluster.cc:1428] Setting key 1dcdd43b610cd3ac0883c462e22d4f35
Sep 26 23:23:38 dist-test-slave-q5g8 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:38.251662 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.228991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58679 (local address 127.0.97.62:42255)
0926 23:23:38.229466 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:38.229471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:38.230593 (+ 1122us) server_negotiation.cc:407] Connection header received
0926 23:23:38.232600 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:38.232607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:38.232730 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:38.232862 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:38.235242 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.236472 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.238446 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.239317 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.239509 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.243612 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:38.243624 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:38.243634 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:38.243690 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:38.247499 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:38.248745 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:38.248767 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:38.248771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:38.248943 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:38.249537 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:38.249541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:38.249544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:38.249915 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:38.250050 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:38.251161 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0926 23:23:38.251383 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":218,"threads_started":1}
I20260926 23:23:38.253461 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.229641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.2:58679)
0926 23:23:38.230389 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:38.230438 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:38.232296 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:38.233204 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:38.233219 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:38.234110 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:38.235041 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.235068 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.236821 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.236827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.238278 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.238291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.239558 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.239563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.239760 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.240557 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:38.240586 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:38.243294 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:38.247694 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:38.247708 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:38.247734 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:38.248503 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:38.249083 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:38.249091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:38.249098 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:38.249387 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:38.250096 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:38.250108 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:38.250646 (+ 538us) client_negotiation.cc:769] Sending connection context
0926 23:23:38.250999 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 23:23:38.251689 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":303,"threads_started":1}
I20260926 23:23:38.255491 4840 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260926 23:23:38.255903 4840 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:38.256784 4840 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260926 23:23:38.258755 4519 ts_manager.cc:194] Registered new tserver with Master: 9e9f8f16db34428db5a11f9016630bbf (127.0.97.2:39119)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:23:38.259981 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:38.288969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:38.641233 4846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:38.641634 4846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:38.641825 4846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:38.650097 4846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:38.650307 4846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:38.650498 4846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:38.661755 4846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:38.663208 4846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:38.664841 4846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:38.678788 4852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:38.678742 4854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:38.678830 4851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:38.679486 4846 server_base.cc:1034] running on GCE node
I20260926 23:23:38.681545 4846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:38.682847 4846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:38.684111 4846 hybrid_clock.cc:648] HybridClock initialized: now 1790465018684054 us; error 54 us; skew 500 ppm
I20260926 23:23:38.684541 4846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:38 dist-test-slave-q5g8 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:38.688185 4846 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:38.689814 4846 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:38.689991 4846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:38.690384 4846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:38.693921 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "522898b337f5437ba83004d247264f77"
format_stamp: "Formatted at 2026-09-26 23:23:38 on dist-test-slave-q5g8"
server_key: "35af301c1792b23c3f3108200a3aa093"
server_key_iv: "e4a79a9426865610ec83449662c6f3a6"
server_key_version: "encryptionkey@0"
I20260926 23:23:38.694891 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "522898b337f5437ba83004d247264f77"
format_stamp: "Formatted at 2026-09-26 23:23:38 on dist-test-slave-q5g8"
server_key: "35af301c1792b23c3f3108200a3aa093"
server_key_iv: "e4a79a9426865610ec83449662c6f3a6"
server_key_version: "encryptionkey@0"
I20260926 23:23:38.700955 4846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:23:38.704910 4860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:38.706449 4846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:38.706651 4846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "522898b337f5437ba83004d247264f77"
format_stamp: "Formatted at 2026-09-26 23:23:38 on dist-test-slave-q5g8"
server_key: "35af301c1792b23c3f3108200a3aa093"
server_key_iv: "e4a79a9426865610ec83449662c6f3a6"
server_key_version: "encryptionkey@0"
I20260926 23:23:38.706892 4846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:38.720361 4707 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
I20260926 23:23:38.731845 4846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:38.736446 4846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:38.736750 4846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:38.737846 4846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:38.739275 4846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:38.739367 4846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:38.739461 4846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:38.739516 4846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:38.775521 4846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42097
I20260926 23:23:38.775573 4972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42097 every 8 connection(s)
I20260926 23:23:38.776419 4846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:38.781678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 4846
I20260926 23:23:38.781859 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:38.782332 388 external_mini_cluster.cc:1428] Setting key 1f851a363db89816151b220a20108ab9
Sep 26 23:23:38 dist-test-slave-q5g8 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:38.796535 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.778936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56997 (local address 127.0.97.62:42255)
0926 23:23:38.779405 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:38.779411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:38.780483 (+ 1072us) server_negotiation.cc:407] Connection header received
0926 23:23:38.782510 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:38.782517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:38.782611 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:38.782745 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:38.784804 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.785676 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.787219 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.787650 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.787745 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.791221 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:38.791227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:38.791231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:38.791269 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:38.793736 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:38.794462 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:38.794466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:38.794469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:38.794584 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:38.794953 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:38.794956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:38.794958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:38.795201 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:38.795308 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:38.795991 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 23:23:38.796204 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260926 23:23:38.797775 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.779612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.3:56997)
0926 23:23:38.780302 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:38.780346 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:38.782197 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:38.783008 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:38.783020 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:38.783781 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:38.784567 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.784591 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.785820 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.785824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.787094 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.787106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.787836 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.787840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.788040 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.788810 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:38.788833 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:38.790993 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:38.793892 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:38.793899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:38.793914 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:38.794349 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:38.794670 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:38.794673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:38.794675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:38.794842 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:38.795299 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:38.795305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:38.795628 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 23:23:38.795849 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 23:23:38.796403 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
I20260926 23:23:38.799299 4973 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260926 23:23:38.799651 4973 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:38.800355 4973 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260926 23:23:38.801892 4519 ts_manager.cc:194] Registered new tserver with Master: 522898b337f5437ba83004d247264f77 (127.0.97.3:42097)
I20260926 23:23:38.802686 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56997
I20260926 23:23:38.810683 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:23:38.830204 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.816234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.0.97.62:42255)
0926 23:23:38.816502 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:38.816513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:38.816921 (+ 408us) server_negotiation.cc:407] Connection header received
0926 23:23:38.817075 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:38.817079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:38.817148 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:38.817235 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:38.818793 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.819570 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.821278 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.821746 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.821904 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.824549 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:38.824557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:38.824570 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:38.824608 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:38.827270 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:38.828044 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:38.828051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:38.828055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:38.828217 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:38.828595 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:38.828600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:38.828603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:38.829019 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:38.829280 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:38.829618 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 23:23:38.829850 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:23:38.834586 4519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:38.841907 4519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:38.874012 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.856621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.97.2:39119)
0926 23:23:38.857546 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:38.857555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:38.858035 (+ 480us) server_negotiation.cc:407] Connection header received
0926 23:23:38.858480 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:38.858493 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:38.858835 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:38.859035 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:38.860211 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.863311 (+ 3100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.870561 (+ 7250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.872249 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.872378 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.872631 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:38.873440 (+ 809us) server_negotiation.cc:299] Negotiation successful
0926 23:23:38.873629 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":780,"threads_started":1}
I20260926 23:23:38.875931 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.856349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39119 (local address 127.0.0.1:50738)
0926 23:23:38.857875 (+ 1526us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:38.857921 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:38.858193 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:38.859429 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:38.859442 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:38.859486 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:38.859980 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.859998 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.863580 (+ 3582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.863589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.870329 (+ 6740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.870349 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.872850 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.872857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.873100 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.873168 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 23:23:38.875257 (+ 2089us) client_negotiation.cc:240] Negotiation successful
0926 23:23:38.875500 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"thread_start_us":1310,"threads_started":1}
I20260926 23:23:38.879429 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.857224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42097 (local address 127.0.0.1:50510)
0926 23:23:38.858174 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:38.858191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:38.858307 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:38.871491 (+ 13184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:38.871497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:38.871532 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:38.872001 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.872014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.874477 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.874485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.876740 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.876753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.878513 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.878521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.878733 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.878767 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:23:38.878875 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 23:23:38.879034 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":681,"threads_started":1}
I20260926 23:23:38.879508 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.857370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.0.97.3:42097)
0926 23:23:38.870499 (+ 13129us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:38.870512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:38.870551 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:23:38.870675 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:38.870684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:38.871010 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:38.872246 (+ 1236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:38.872597 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.874305 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.876936 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.878247 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.878407 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.878665 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:38.878942 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 23:23:38.879144 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12898,"thread_start_us":13829,"threads_started":1}
I20260926 23:23:38.880735 4775 tablet_service.cc:1467] Processing CreateTablet for tablet fb94b7c8f32d444087fae33b4dd0e5be (DEFAULT_TABLE table=test-table [id=b9a8464bb0e34a728af01c745b22c26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:38.882362 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb94b7c8f32d444087fae33b4dd0e5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:38.883679 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.858538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45917 (local address 127.0.0.1:37120)
0926 23:23:38.874065 (+ 15527us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:38.874094 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:38.874287 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:38.875819 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:38.875825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:38.875852 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:38.876307 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.876327 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.878087 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.878093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.880641 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:38.880661 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.882852 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.882859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:38.883043 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.883080 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:23:38.883188 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 23:23:38.883338 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15332,"thread_start_us":16317,"threads_started":1}
I20260926 23:23:38.883769 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:38.858946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.0.97.1:45917)
0926 23:23:38.874929 (+ 15983us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:38.874939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:38.874973 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:23:38.875076 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:38.875087 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:38.875498 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:38.875723 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:38.876527 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.877923 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.880886 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:38.882629 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:38.882782 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:38.883031 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:38.883257 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 23:23:38.883429 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15790,"thread_start_us":15621,"threads_started":1}
I20260926 23:23:38.883770 4908 tablet_service.cc:1467] Processing CreateTablet for tablet fb94b7c8f32d444087fae33b4dd0e5be (DEFAULT_TABLE table=test-table [id=b9a8464bb0e34a728af01c745b22c26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:38.885394 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb94b7c8f32d444087fae33b4dd0e5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:38.888011 4642 tablet_service.cc:1467] Processing CreateTablet for tablet fb94b7c8f32d444087fae33b4dd0e5be (DEFAULT_TABLE table=test-table [id=b9a8464bb0e34a728af01c745b22c26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:38.889606 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb94b7c8f32d444087fae33b4dd0e5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:38.894905 4992 tablet_bootstrap.cc:492] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Bootstrap starting.
I20260926 23:23:38.898887 4992 tablet_bootstrap.cc:654] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:38.900064 4992 log.cc:824] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:38.901192 4993 tablet_bootstrap.cc:492] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Bootstrap starting.
I20260926 23:23:38.905187 4992 tablet_bootstrap.cc:492] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: No bootstrap required, opened a new log
I20260926 23:23:38.905548 4992 ts_tablet_manager.cc:1397] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 23:23:38.905903 4993 tablet_bootstrap.cc:654] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:38.905988 4995 tablet_bootstrap.cc:492] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Bootstrap starting.
I20260926 23:23:38.907473 4993 log.cc:824] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:38.909628 4995 tablet_bootstrap.cc:654] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:38.910912 4995 log.cc:824] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:38.914248 4995 tablet_bootstrap.cc:492] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: No bootstrap required, opened a new log
I20260926 23:23:38.914546 4995 ts_tablet_manager.cc:1397] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 23:23:38.918011 4993 tablet_bootstrap.cc:492] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: No bootstrap required, opened a new log
I20260926 23:23:38.918339 4993 ts_tablet_manager.cc:1397] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.004s
I20260926 23:23:38.918224 4992 raft_consensus.cc:348] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:38.918754 4992 raft_consensus.cc:374] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:38.919209 4992 raft_consensus.cc:729] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9f8f16db34428db5a11f9016630bbf, State: Initialized, Role: FOLLOWER
I20260926 23:23:38.919966 4992 consensus_queue.cc:260] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:38.921284 4840 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
I20260926 23:23:38.922088 4992 ts_tablet_manager.cc:1428] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260926 23:23:38.925318 4995 raft_consensus.cc:348] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:38.925848 4995 raft_consensus.cc:374] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:38.925971 4995 raft_consensus.cc:729] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf40c915cd2d44ba81a6065b0c9b9384, State: Initialized, Role: FOLLOWER
I20260926 23:23:38.926709 4995 consensus_queue.cc:260] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:38.928901 4995 ts_tablet_manager.cc:1428] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 23:23:38.930660 4993 raft_consensus.cc:348] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:38.931241 4993 raft_consensus.cc:374] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:38.931366 4993 raft_consensus.cc:729] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522898b337f5437ba83004d247264f77, State: Initialized, Role: FOLLOWER
I20260926 23:23:38.932022 4993 consensus_queue.cc:260] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:38.933372 4973 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
W20260926 23:23:38.933773 4708 tablet.cc:2369] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:38.934122 4993 ts_tablet_manager.cc:1428] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
W20260926 23:23:38.978547 4841 tablet.cc:2369] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:23:39.028141 4974 tablet.cc:2369] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:39.103099 5000 raft_consensus.cc:482] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:39.103469 5000 raft_consensus.cc:504] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:39.105682 5000 leader_election.cc:288] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9f8f16db34428db5a11f9016630bbf (127.0.97.2:39119), cf40c915cd2d44ba81a6065b0c9b9384 (127.0.97.1:45917)
I20260926 23:23:39.113866 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.106085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46043 (local address 127.0.97.2:39119)
0926 23:23:39.106329 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:39.106338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:39.106704 (+ 366us) server_negotiation.cc:407] Connection header received
0926 23:23:39.106958 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:39.106964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:39.107033 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:39.107127 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:39.107502 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.108575 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.111324 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.112729 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.112912 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.113035 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:39.113377 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 23:23:39.113548 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:23:39.113879 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.106118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39119 (local address 127.0.97.3:46043)
0926 23:23:39.106566 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:39.106593 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:39.106773 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:39.107148 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:39.107154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:39.107183 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:39.107360 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:39.107367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.108771 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.108775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:39.111043 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:39.111068 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.112913 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.112917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:39.113130 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.113187 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 23:23:39.113305 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 23:23:39.113517 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":89}
I20260926 23:23:39.114221 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.106215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45917 (local address 127.0.97.3:57069)
0926 23:23:39.106933 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:39.106957 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:39.107095 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:39.107525 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:39.107529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:39.107558 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:39.107833 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:39.107840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.109162 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.109167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:39.111481 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:39.111492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.113275 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.113283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:39.113489 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.113518 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 23:23:39.113607 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 23:23:39.113741 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":68,"thread_start_us":376,"threads_started":1}
I20260926 23:23:39.114655 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.106160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57069 (local address 127.0.97.1:45917)
0926 23:23:39.106366 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:39.106374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:39.107045 (+ 671us) server_negotiation.cc:407] Connection header received
0926 23:23:39.107271 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:39.107277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:39.107364 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:39.107497 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:39.108009 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.109016 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.111645 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.113049 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.113196 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.113310 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:39.113661 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 23:23:39.113813 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:23:39.115067 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb94b7c8f32d444087fae33b4dd0e5be" candidate_uuid: "522898b337f5437ba83004d247264f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9f8f16db34428db5a11f9016630bbf" is_pre_election: true
I20260926 23:23:39.115721 4795 raft_consensus.cc:2454] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522898b337f5437ba83004d247264f77 in term 0.
I20260926 23:23:39.115658 4662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb94b7c8f32d444087fae33b4dd0e5be" candidate_uuid: "522898b337f5437ba83004d247264f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" is_pre_election: true
I20260926 23:23:39.116191 4662 raft_consensus.cc:2454] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522898b337f5437ba83004d247264f77 in term 0.
I20260926 23:23:39.116580 4862 leader_election.cc:302] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522898b337f5437ba83004d247264f77, 9e9f8f16db34428db5a11f9016630bbf; no voters:
I20260926 23:23:39.117086 5000 raft_consensus.cc:2781] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:39.117242 5000 raft_consensus.cc:482] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:39.117374 5000 raft_consensus.cc:3037] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:39.119616 5000 raft_consensus.cc:504] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:39.120252 5000 leader_election.cc:288] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [CANDIDATE]: Term 1 election: Requested vote from peers 9e9f8f16db34428db5a11f9016630bbf (127.0.97.2:39119), cf40c915cd2d44ba81a6065b0c9b9384 (127.0.97.1:45917)
I20260926 23:23:39.120888 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb94b7c8f32d444087fae33b4dd0e5be" candidate_uuid: "522898b337f5437ba83004d247264f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9f8f16db34428db5a11f9016630bbf"
I20260926 23:23:39.120878 4662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb94b7c8f32d444087fae33b4dd0e5be" candidate_uuid: "522898b337f5437ba83004d247264f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf40c915cd2d44ba81a6065b0c9b9384"
I20260926 23:23:39.121114 4662 raft_consensus.cc:3037] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:39.121114 4795 raft_consensus.cc:3037] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:39.124461 4795 raft_consensus.cc:2454] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522898b337f5437ba83004d247264f77 in term 1.
I20260926 23:23:39.124461 4662 raft_consensus.cc:2454] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522898b337f5437ba83004d247264f77 in term 1.
I20260926 23:23:39.125007 4862 leader_election.cc:302] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 522898b337f5437ba83004d247264f77, 9e9f8f16db34428db5a11f9016630bbf, cf40c915cd2d44ba81a6065b0c9b9384; no voters:
I20260926 23:23:39.125435 5000 raft_consensus.cc:2781] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:39.125918 5000 raft_consensus.cc:686] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 1 LEADER]: Becoming Leader. State: Replica: 522898b337f5437ba83004d247264f77, State: Running, Role: LEADER
I20260926 23:23:39.126523 5000 consensus_queue.cc:237] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } }
I20260926 23:23:39.131232 4519 catalog_manager.cc:5495] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 522898b337f5437ba83004d247264f77 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "522898b337f5437ba83004d247264f77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522898b337f5437ba83004d247264f77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 } health_report { overall_health: UNKNOWN } } }
I20260926 23:23:39.168092 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.162035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.0.97.3:42097)
0926 23:23:39.162300 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:39.162307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:39.162440 (+ 133us) server_negotiation.cc:407] Connection header received
0926 23:23:39.162713 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:39.162717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:39.162809 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:39.162980 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:39.163394 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.164199 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.165737 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.166159 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.166308 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.166875 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:39.167433 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:39.167605 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:39.167716 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 23:23:39.167806 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 23:23:39.178323 4662 raft_consensus.cc:1261] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 1 FOLLOWER]: Refusing update from remote peer 522898b337f5437ba83004d247264f77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:39.178325 4795 raft_consensus.cc:1261] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 1 FOLLOWER]: Refusing update from remote peer 522898b337f5437ba83004d247264f77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:39.179533 5002 consensus_queue.cc:1035] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf40c915cd2d44ba81a6065b0c9b9384" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:39.180011 5000 consensus_queue.cc:1035] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9f8f16db34428db5a11f9016630bbf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:39.192600 5005 mvcc.cc:205] Tried to move back new op lower bound from 7333744718542307328 to 7333744718346518528. Current Snapshot: MvccSnapshot[applied={T|T < 7333744718542307328}]
I20260926 23:23:39.210126 4518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:39.210526 4518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:39.214896 4518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2464649de6c744c79c003a82337632c9: Sending DeleteTablet for 3 replicas of tablet fb94b7c8f32d444087fae33b4dd0e5be
I20260926 23:23:39.216500 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet fb94b7c8f32d444087fae33b4dd0e5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50738
I20260926 23:23:39.216481 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet fb94b7c8f32d444087fae33b4dd0e5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37120
I20260926 23:23:39.217366 4908 tablet_service.cc:1514] Processing DeleteTablet for tablet fb94b7c8f32d444087fae33b4dd0e5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50510
I20260926 23:23:39.218091 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 4580
I20260926 23:23:39.242486 5018 tablet_replica.cc:330] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: stopping tablet replica
I20260926 23:23:39.243011 5018 raft_consensus.cc:2229] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:39.244546 5018 raft_consensus.cc:2258] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:39.245218 5016 tablet_replica.cc:330] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: stopping tablet replica
I20260926 23:23:39.245805 5017 tablet_replica.cc:330] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: stopping tablet replica
I20260926 23:23:39.246183 5017 raft_consensus.cc:2229] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:39.246672 5017 raft_consensus.cc:2258] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:39.245639 5016 raft_consensus.cc:2229] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:39.247310 5016 raft_consensus.cc:2258] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 23:23:39 dist-test-slave-q5g8 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:39.250448 5017 ts_tablet_manager.cc:1905] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:39.251833 5018 ts_tablet_manager.cc:1905] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:39.255198 5016 ts_tablet_manager.cc:1905] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:39.259570 5017 ts_tablet_manager.cc:1918] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:39.259778 5017 log.cc:1196] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/fb94b7c8f32d444087fae33b4dd0e5be
I20260926 23:23:39.260396 5017 ts_tablet_manager.cc:1939] T fb94b7c8f32d444087fae33b4dd0e5be P 9e9f8f16db34428db5a11f9016630bbf: Deleting consensus metadata
I20260926 23:23:39.263090 4504 catalog_manager.cc:4841] TS 9e9f8f16db34428db5a11f9016630bbf (127.0.97.2:39119): tablet fb94b7c8f32d444087fae33b4dd0e5be (table test-table [id=b9a8464bb0e34a728af01c745b22c26a]) successfully deleted
I20260926 23:23:39.266410 5018 ts_tablet_manager.cc:1918] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:39.266726 5018 log.cc:1196] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/fb94b7c8f32d444087fae33b4dd0e5be
I20260926 23:23:39.267439 5018 ts_tablet_manager.cc:1939] T fb94b7c8f32d444087fae33b4dd0e5be P 522898b337f5437ba83004d247264f77: Deleting consensus metadata
I20260926 23:23:39.267534 5016 ts_tablet_manager.cc:1918] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:39.267714 5016 log.cc:1196] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/fb94b7c8f32d444087fae33b4dd0e5be
I20260926 23:23:39.268249 5016 ts_tablet_manager.cc:1939] T fb94b7c8f32d444087fae33b4dd0e5be P cf40c915cd2d44ba81a6065b0c9b9384: Deleting consensus metadata
I20260926 23:23:39.269887 4503 catalog_manager.cc:4841] TS 522898b337f5437ba83004d247264f77 (127.0.97.3:42097): tablet fb94b7c8f32d444087fae33b4dd0e5be (table test-table [id=b9a8464bb0e34a728af01c745b22c26a]) successfully deleted
I20260926 23:23:39.270332 4503 catalog_manager.cc:4841] TS cf40c915cd2d44ba81a6065b0c9b9384 (127.0.97.1:45917): tablet fb94b7c8f32d444087fae33b4dd0e5be (table test-table [id=b9a8464bb0e34a728af01c745b22c26a]) successfully deleted
I20260926 23:23:39.271450 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.219683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.0.97.1:45917)
0926 23:23:39.219944 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:39.219954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:39.230879 (+ 10925us) server_negotiation.cc:407] Connection header received
0926 23:23:39.231444 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:39.231451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:39.231554 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:39.231626 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:39.233563 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.234839 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.241299 (+ 6460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.242103 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.242223 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.254056 (+ 11833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:39.254079 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:39.254103 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:39.254165 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:39.258883 (+ 4718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:39.260158 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:39.260175 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:39.260179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:39.260399 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:39.261812 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:39.261822 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:39.261827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:39.262427 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:39.262766 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:39.270740 (+ 7974us) server_negotiation.cc:299] Negotiation successful
0926 23:23:39.271029 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 23:23:39.272209 4702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:39.386054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 4580
I20260926 23:23:39.404474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 4713
Sep 26 23:23:39 dist-test-slave-q5g8 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:39.423481 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.405556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.0.97.2:39119)
0926 23:23:39.405790 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:39.405797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:39.405833 (+ 36us) server_negotiation.cc:407] Connection header received
0926 23:23:39.405911 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:39.405917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:39.406008 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:39.406140 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:39.407454 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.408959 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.410393 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.411122 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.411326 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.416083 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:39.416105 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:39.416129 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:39.416195 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:39.420038 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:39.421091 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:39.421103 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:39.421107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:39.421289 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:39.421669 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:39.421674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:39.421677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:39.422186 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:39.422519 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:39.422843 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 23:23:39.423089 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:23:39.424232 4835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:39.517925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 4713
I20260926 23:23:39.535635 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 4846
Sep 26 23:23:39 dist-test-slave-q5g8 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:39.555060 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:39.536486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.0.97.3:42097)
0926 23:23:39.536653 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:39.536658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:39.536748 (+ 90us) server_negotiation.cc:407] Connection header received
0926 23:23:39.536998 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:39.537005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:39.537067 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:39.537148 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:39.538957 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.539771 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.541771 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:39.542225 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:39.542344 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:39.548005 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:39.548020 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:39.548038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:39.548091 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:39.551759 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:39.552568 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:39.552579 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:39.552582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:39.552825 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:39.553260 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:39.553265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:39.553269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:39.553749 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:39.554124 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:39.554424 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 23:23:39.554665 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:23:39.555910 4968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:39.680322 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 4846
I20260926 23:23:39.700735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 4488
I20260926 23:23:39.701980 4549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:39.813887 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 4488
2026-09-26T23:23:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4604 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T23:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:39Z Disabled control of system clock
I20260926 23:23:39.886032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33369
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 23:23:40.233731 5027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:40.234046 5027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:40.234228 5027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:40.242535 5027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:40.242703 5027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:40.242779 5027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:40.242865 5027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:40.242934 5027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:40.254128 5027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:40.255780 5027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:40.257984 5027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:40.270820 5035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:40.270874 5033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:40.270851 5032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:40.271612 5027 server_base.cc:1034] running on GCE node
I20260926 23:23:40.272478 5027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:40.274011 5027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:40.275264 5027 hybrid_clock.cc:648] HybridClock initialized: now 1790465020275218 us; error 45 us; skew 500 ppm
I20260926 23:23:40.275620 5027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:40.277938 5027 webserver.cc:460] Webserver started at http://127.0.97.62:32847/ using document root <none> and password file <none>
I20260926 23:23:40.278662 5027 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:40.278798 5027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:40.279131 5027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:40.282161 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "29fb3fa8bdfd4e8297aa45140e0e840a"
format_stamp: "Formatted at 2026-09-26 23:23:40 on dist-test-slave-q5g8"
server_key: "3c7ef50247c4f3b707b203f535db3ef5"
server_key_iv: "8d4877cac60ea3d3db5ba222914f497b"
server_key_version: "encryptionkey@0"
I20260926 23:23:40.283003 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "29fb3fa8bdfd4e8297aa45140e0e840a"
format_stamp: "Formatted at 2026-09-26 23:23:40 on dist-test-slave-q5g8"
server_key: "3c7ef50247c4f3b707b203f535db3ef5"
server_key_iv: "8d4877cac60ea3d3db5ba222914f497b"
server_key_version: "encryptionkey@0"
I20260926 23:23:40.288652 5027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:23:40.292544 5041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:40.294137 5027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:23:40.294368 5027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "29fb3fa8bdfd4e8297aa45140e0e840a"
format_stamp: "Formatted at 2026-09-26 23:23:40 on dist-test-slave-q5g8"
server_key: "3c7ef50247c4f3b707b203f535db3ef5"
server_key_iv: "8d4877cac60ea3d3db5ba222914f497b"
server_key_version: "encryptionkey@0"
I20260926 23:23:40.294610 5027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:40.301043 5027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:40.302009 5027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:40.302345 5027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:40.323683 5027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33369
I20260926 23:23:40.323716 5092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33369 every 8 connection(s)
I20260926 23:23:40.325394 5027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:40.327962 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5027
I20260926 23:23:40.328203 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:40.329057 388 external_mini_cluster.cc:1428] Setting key 1654df286deed99d2d9829df1ff114df
I20260926 23:23:40.330456 5093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:40.340212 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:40.331682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.0.97.62:33369)
0926 23:23:40.333274 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:40.333288 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:40.333424 (+ 136us) server_negotiation.cc:407] Connection header received
0926 23:23:40.335970 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:40.336001 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:40.336575 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:40.337076 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:40.338358 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:40.338365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:40.338378 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:40.338394 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:40.338464 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:40.338485 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:40.338698 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:40.338815 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 23:23:40.339178 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":542,"threads_started":1}
I20260926 23:23:40.345407 5093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a: Bootstrap starting.
I20260926 23:23:40.348875 5093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:40.349960 5093 log.cc:824] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:40.352861 5093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a: No bootstrap required, opened a new log
I20260926 23:23:40.360577 5093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33369 } }
I20260926 23:23:40.361063 5093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:40.361153 5093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fb3fa8bdfd4e8297aa45140e0e840a, State: Initialized, Role: FOLLOWER
I20260926 23:23:40.361887 5093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33369 } }
I20260926 23:23:40.362186 5093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:40.362299 5093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:40.362433 5093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:40.365144 5093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33369 } }
I20260926 23:23:40.365639 5093 leader_election.cc:302] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29fb3fa8bdfd4e8297aa45140e0e840a; no voters:
I20260926 23:23:40.366290 5093 leader_election.cc:288] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:40.366492 5098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:40.367307 5098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [term 1 LEADER]: Becoming Leader. State: Replica: 29fb3fa8bdfd4e8297aa45140e0e840a, State: Running, Role: LEADER
I20260926 23:23:40.367905 5098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33369 } }
I20260926 23:23:40.368587 5093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:40.371165 5099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33369 } } }
I20260926 23:23:40.371682 5099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:40.371165 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29fb3fa8bdfd4e8297aa45140e0e840a. Latest consensus state: current_term: 1 leader_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fb3fa8bdfd4e8297aa45140e0e840a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33369 } } }
I20260926 23:23:40.373936 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:40.380383 5104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:40.387477 5104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:40.396952 5104 catalog_manager.cc:1347] Generated new cluster ID: cbdfb160d70d4e45a62c35d128f04f2e
I20260926 23:23:40.397097 5104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:40.423908 5104 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:40.424489 5104 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:40.434733 5104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a: Generated new TSK 0
I20260926 23:23:40.435897 5104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:23:40.443427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33369
--builtin_ntp_servers=127.0.97.20:36481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 23:23:40.807317 5117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:40.807650 5117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:40.807751 5117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:40.815438 5117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:40.815583 5117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:40.815707 5117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:40.825377 5117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:40.826746 5117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:40.828624 5117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:40.843894 5122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:40.843847 5123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:40.843801 5125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:40.845505 5117 server_base.cc:1034] running on GCE node
I20260926 23:23:40.846282 5117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:40.847370 5117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:40.848774 5117 hybrid_clock.cc:648] HybridClock initialized: now 1790465020848642 us; error 55 us; skew 500 ppm
I20260926 23:23:40.849227 5117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:40.851871 5117 webserver.cc:460] Webserver started at http://127.0.97.1:46119/ using document root <none> and password file <none>
I20260926 23:23:40.852669 5117 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:40.852880 5117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:40.853227 5117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:40.856369 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6"
format_stamp: "Formatted at 2026-09-26 23:23:40 on dist-test-slave-q5g8"
server_key: "59f17fb074dd1281e26849c3ec1a33fa"
server_key_iv: "423c19d5ae91959735d8e314c2ed66e9"
server_key_version: "encryptionkey@0"
I20260926 23:23:40.857347 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6"
format_stamp: "Formatted at 2026-09-26 23:23:40 on dist-test-slave-q5g8"
server_key: "59f17fb074dd1281e26849c3ec1a33fa"
server_key_iv: "423c19d5ae91959735d8e314c2ed66e9"
server_key_version: "encryptionkey@0"
I20260926 23:23:40.874505 5117 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.012s sys 0.004s
I20260926 23:23:40.878633 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:40.880113 5117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:23:40.880416 5117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6"
format_stamp: "Formatted at 2026-09-26 23:23:40 on dist-test-slave-q5g8"
server_key: "59f17fb074dd1281e26849c3ec1a33fa"
server_key_iv: "423c19d5ae91959735d8e314c2ed66e9"
server_key_version: "encryptionkey@0"
I20260926 23:23:40.880733 5117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:40.886933 5117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:40.887833 5117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:40.888142 5117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:40.889361 5117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:40.890980 5117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:40.891124 5117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:40.891263 5117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:40.891359 5117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:40.929862 5117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43401
I20260926 23:23:40.929921 5243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43401 every 8 connection(s)
I20260926 23:23:40.931491 5117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:40.935941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5117
I20260926 23:23:40.936223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:40.937268 388 external_mini_cluster.cc:1428] Setting key 73db559a5ef738abc84263e9c63019d0
I20260926 23:23:40.940820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33369
--builtin_ntp_servers=127.0.97.20:36481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 23:23:40.941812 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:40.934430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37941 (local address 127.0.97.62:33369)
0926 23:23:40.934970 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:40.934979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:40.936766 (+ 1787us) server_negotiation.cc:407] Connection header received
0926 23:23:40.939001 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:40.939007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:40.939112 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:40.939245 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:40.941037 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:40.941046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:40.941053 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:40.941072 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:40.941133 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:40.941155 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:40.941259 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:40.941542 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 23:23:40.941593 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
I20260926 23:23:40.943399 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:40.935002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33369 (local address 127.0.97.1:37941)
0926 23:23:40.936273 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:40.936338 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:40.938701 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:40.939576 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:40.939589 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:40.939730 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:40.940543 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:40.940550 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:40.940612 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:40.940799 (+ 187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:40.941272 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:40.941289 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:40.941301 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 23:23:40.941643 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 23:23:40.941956 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":325,"threads_started":1}
I20260926 23:23:40.945243 5244 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33369
I20260926 23:23:40.945578 5244 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:40.946216 5244 heartbeater.cc:511] Master 127.0.97.62:33369 requested a full tablet report, sending...
I20260926 23:23:40.948629 5058 ts_manager.cc:194] Registered new tserver with Master: 6acf6e3cf9c64967b3bd381fa7e46fe6 (127.0.97.1:43401)
W20260926 23:23:41.293690 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:41.293992 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:41.294145 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:41.301816 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:41.301926 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:41.302096 5248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:41.311419 5248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:41.312865 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:41.314599 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:41.329783 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:41.330025 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:41.329741 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:41.332697 5248 server_base.cc:1034] running on GCE node
I20260926 23:23:41.333503 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:41.334707 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:41.335953 5248 hybrid_clock.cc:648] HybridClock initialized: now 1790465021335907 us; error 41 us; skew 500 ppm
I20260926 23:23:41.336330 5248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:41.339293 5248 webserver.cc:460] Webserver started at http://127.0.97.2:36775/ using document root <none> and password file <none>
I20260926 23:23:41.340293 5248 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:41.340417 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:41.340812 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:41.344794 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "d7258cc7838c465586f251f6e05f0431"
format_stamp: "Formatted at 2026-09-26 23:23:41 on dist-test-slave-q5g8"
server_key: "ff577fa48804b8fe9778daae732412e5"
server_key_iv: "6b9b1ee418f8712d0076a7432e89be5a"
server_key_version: "encryptionkey@0"
I20260926 23:23:41.345682 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d7258cc7838c465586f251f6e05f0431"
format_stamp: "Formatted at 2026-09-26 23:23:41 on dist-test-slave-q5g8"
server_key: "ff577fa48804b8fe9778daae732412e5"
server_key_iv: "6b9b1ee418f8712d0076a7432e89be5a"
server_key_version: "encryptionkey@0"
I20260926 23:23:41.351246 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 23:23:41.355103 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:41.356997 5248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:41.357223 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "d7258cc7838c465586f251f6e05f0431"
format_stamp: "Formatted at 2026-09-26 23:23:41 on dist-test-slave-q5g8"
server_key: "ff577fa48804b8fe9778daae732412e5"
server_key_iv: "6b9b1ee418f8712d0076a7432e89be5a"
server_key_version: "encryptionkey@0"
I20260926 23:23:41.357509 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:41.364259 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:41.365302 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:41.365623 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:41.366726 5248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:41.368408 5248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:41.368549 5248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:41.368769 5248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:41.368883 5248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:41.409298 5248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46089
I20260926 23:23:41.409363 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46089 every 8 connection(s)
I20260926 23:23:41.410813 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:41.414788 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5248
I20260926 23:23:41.415006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:41.415441 388 external_mini_cluster.cc:1428] Setting key d57d558ea22e92d4bd52f084590e38cf
I20260926 23:23:41.418390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33369
--builtin_ntp_servers=127.0.97.20:36481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 23:23:41.420492 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.413865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42225 (local address 127.0.97.62:33369)
0926 23:23:41.414089 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:41.414095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:41.415847 (+ 1752us) server_negotiation.cc:407] Connection header received
0926 23:23:41.418001 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:41.418006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:41.418083 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:41.418210 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:41.419701 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:41.419709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:41.419716 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:41.419740 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:41.419783 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:41.419802 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:41.419941 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:41.420097 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 23:23:41.420162 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:23:41.421416 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.414437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33369 (local address 127.0.97.2:42225)
0926 23:23:41.415652 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:41.415705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:41.417699 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:41.418427 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:41.418435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:41.418596 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:41.419384 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:41.419387 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:41.419427 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:41.419519 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:41.419902 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:41.419909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:41.419914 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:23:41.420131 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 23:23:41.420338 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":390,"threads_started":1}
I20260926 23:23:41.422889 5375 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33369
I20260926 23:23:41.423233 5375 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:41.423946 5375 heartbeater.cc:511] Master 127.0.97.62:33369 requested a full tablet report, sending...
I20260926 23:23:41.425493 5058 ts_manager.cc:194] Registered new tserver with Master: d7258cc7838c465586f251f6e05f0431 (127.0.97.2:46089)
W20260926 23:23:41.750427 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:41.750766 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:41.750949 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:41.758319 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:41.758448 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:41.758631 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:41.768013 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:41.769495 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:41.771420 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:41.785095 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:41.785156 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:41.785351 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:41.785859 5378 server_base.cc:1034] running on GCE node
I20260926 23:23:41.786512 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:41.787467 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:41.788729 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790465021788629 us; error 56 us; skew 500 ppm
I20260926 23:23:41.789052 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:41.791224 5378 webserver.cc:460] Webserver started at http://127.0.97.3:38509/ using document root <none> and password file <none>
I20260926 23:23:41.791923 5378 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:41.792052 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:41.792361 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:41.795212 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "dce70ba69db747d6a0e850c855211e4b"
format_stamp: "Formatted at 2026-09-26 23:23:41 on dist-test-slave-q5g8"
server_key: "26fdd63d579e5bdf1286f99fe453d2aa"
server_key_iv: "f6df8f225539db627b57121c52837ac7"
server_key_version: "encryptionkey@0"
I20260926 23:23:41.796001 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dce70ba69db747d6a0e850c855211e4b"
format_stamp: "Formatted at 2026-09-26 23:23:41 on dist-test-slave-q5g8"
server_key: "26fdd63d579e5bdf1286f99fe453d2aa"
server_key_iv: "f6df8f225539db627b57121c52837ac7"
server_key_version: "encryptionkey@0"
I20260926 23:23:41.801291 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:23:41.804631 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:41.806043 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:41.806217 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "dce70ba69db747d6a0e850c855211e4b"
format_stamp: "Formatted at 2026-09-26 23:23:41 on dist-test-slave-q5g8"
server_key: "26fdd63d579e5bdf1286f99fe453d2aa"
server_key_iv: "f6df8f225539db627b57121c52837ac7"
server_key_version: "encryptionkey@0"
I20260926 23:23:41.806450 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:41.812505 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:41.813374 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:41.813658 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:41.814594 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:41.815976 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:41.816089 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:41.816216 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:41.816308 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:41.848344 5378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44365
I20260926 23:23:41.848392 5504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44365 every 8 connection(s)
I20260926 23:23:41.849787 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:41.858434 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.852553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45911 (local address 127.0.97.62:33369)
0926 23:23:41.852784 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:41.852789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:41.854446 (+ 1657us) server_negotiation.cc:407] Connection header received
0926 23:23:41.855986 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:41.855992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:41.856062 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:41.856185 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:41.857611 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:41.857617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:41.857621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:41.857643 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:41.857685 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:41.857698 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:41.857824 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:41.858070 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 23:23:41.858141 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:23:41.859203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5378
I20260926 23:23:41.859390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:41.859434 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.853137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33369 (local address 127.0.97.3:45911)
0926 23:23:41.854052 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:41.854107 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:41.855714 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:41.856411 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:41.856419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:41.856516 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:41.857289 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:41.857292 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:41.857331 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:41.857420 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:41.857807 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:41.857817 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:41.857825 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:23:41.858157 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 23:23:41.858439 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
I20260926 23:23:41.859977 388 external_mini_cluster.cc:1428] Setting key 0cd7fc177db471f538acd3b5ce79f880
I20260926 23:23:41.861123 5505 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33369
I20260926 23:23:41.861496 5505 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:41.862205 5505 heartbeater.cc:511] Master 127.0.97.62:33369 requested a full tablet report, sending...
I20260926 23:23:41.863689 5058 ts_manager.cc:194] Registered new tserver with Master: dce70ba69db747d6a0e850c855211e4b (127.0.97.3:44365)
I20260926 23:23:41.875411 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:23:41.882596 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.880095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.0.97.62:33369)
0926 23:23:41.880302 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:41.880309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:41.880820 (+ 511us) server_negotiation.cc:407] Connection header received
0926 23:23:41.880967 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:41.880970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:41.881024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:41.881116 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:41.882018 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:41.882021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:41.882024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:41.882038 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:41.882069 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:41.882082 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:41.882179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:41.882243 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 23:23:41.882285 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:23:41.886613 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:41.894215 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:41.913213 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.906839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44365 (local address 127.0.0.1:52900)
0926 23:23:41.908548 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:41.908607 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:41.908903 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:41.910942 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:41.910948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:41.911035 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:41.911880 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:41.911882 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:41.911903 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:41.912006 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:41.912393 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:41.912402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:41.912410 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:23:41.912632 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 23:23:41.912750 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"thread_start_us":2176,"threads_started":1}
I20260926 23:23:41.914359 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.907017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.0.97.3:44365)
0926 23:23:41.907845 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:41.907857 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:41.908764 (+ 907us) server_negotiation.cc:407] Connection header received
0926 23:23:41.909813 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:41.909827 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:41.910165 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:41.910321 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:41.912166 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:41.912176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:41.912190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:41.912210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:41.912275 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:41.912297 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:41.912460 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:41.913846 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0926 23:23:41.913963 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":511,"threads_started":1}
I20260926 23:23:41.914603 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.908187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.0.97.2:46089)
0926 23:23:41.911929 (+ 3742us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:41.911942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:41.911972 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:23:41.912041 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:41.912046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:41.912252 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:41.912402 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:41.913828 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:41.913839 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:41.913853 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:41.913876 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:41.913933 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:41.913950 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:41.914127 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:41.914178 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 23:23:41.914263 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3554,"thread_start_us":3419,"threads_started":1}
I20260926 23:23:41.914652 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.907562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46089 (local address 127.0.0.1:56398)
0926 23:23:41.909308 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:41.909325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:41.909410 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:41.912406 (+ 2996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:41.912409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:41.912439 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:41.913234 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:41.913239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:41.913256 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:41.913298 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:41.914053 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:41.914058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:41.914064 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:23:41.914201 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 23:23:41.914255 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1598,"thread_start_us":539,"threads_started":1}
I20260926 23:23:41.916070 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.909892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.0.97.1:43401)
0926 23:23:41.910595 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:41.910604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:41.910632 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:23:41.910696 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:41.910704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:41.911004 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:41.912173 (+ 1169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:41.915277 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:41.915284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:41.915296 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:41.915317 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:41.915362 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:41.915380 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:41.915519 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:41.915635 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 23:23:41.915726 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":752,"threads_started":1}
I20260926 23:23:41.916080 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:41.909362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39620)
0926 23:23:41.909911 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:41.909926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:41.910010 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:41.911194 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:41.911199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:41.911223 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:41.912606 (+ 1383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:41.912609 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:41.912624 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:41.912673 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:41.915496 (+ 2823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:41.915502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:41.915508 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:23:41.915651 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:23:41.915702 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":748,"threads_started":1}
I20260926 23:23:41.918423 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 50446ac32cad4e0aa126cd53cf9726f5 (DEFAULT_TABLE table=test-table [id=e61bb02a25c1485fbc442a3f1a0e480d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:41.918501 5310 tablet_service.cc:1467] Processing CreateTablet for tablet 50446ac32cad4e0aa126cd53cf9726f5 (DEFAULT_TABLE table=test-table [id=e61bb02a25c1485fbc442a3f1a0e480d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:41.919725 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50446ac32cad4e0aa126cd53cf9726f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:41.919838 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50446ac32cad4e0aa126cd53cf9726f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:41.919678 5179 tablet_service.cc:1467] Processing CreateTablet for tablet 50446ac32cad4e0aa126cd53cf9726f5 (DEFAULT_TABLE table=test-table [id=e61bb02a25c1485fbc442a3f1a0e480d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:41.920965 5179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50446ac32cad4e0aa126cd53cf9726f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:41.938131 5523 tablet_bootstrap.cc:492] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Bootstrap starting.
I20260926 23:23:41.938127 5524 tablet_bootstrap.cc:492] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Bootstrap starting.
I20260926 23:23:41.942659 5523 tablet_bootstrap.cc:654] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:41.944029 5523 log.cc:824] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:41.944664 5524 tablet_bootstrap.cc:654] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:41.946110 5525 tablet_bootstrap.cc:492] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6: Bootstrap starting.
I20260926 23:23:41.946099 5524 log.cc:824] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:41.947767 5523 tablet_bootstrap.cc:492] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: No bootstrap required, opened a new log
I20260926 23:23:41.948132 5523 ts_tablet_manager.cc:1397] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260926 23:23:41.950522 5525 tablet_bootstrap.cc:654] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:41.951611 5524 tablet_bootstrap.cc:492] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: No bootstrap required, opened a new log
I20260926 23:23:41.951689 5244 heartbeater.cc:503] Master 127.0.97.62:33369 was elected leader, sending a full tablet report...
I20260926 23:23:41.951812 5525 log.cc:824] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:41.951891 5524 ts_tablet_manager.cc:1397] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260926 23:23:41.958069 5523 raft_consensus.cc:348] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:41.958633 5523 raft_consensus.cc:374] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:41.958755 5523 raft_consensus.cc:729] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce70ba69db747d6a0e850c855211e4b, State: Initialized, Role: FOLLOWER
I20260926 23:23:41.959393 5523 consensus_queue.cc:260] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:41.960595 5505 heartbeater.cc:503] Master 127.0.97.62:33369 was elected leader, sending a full tablet report...
I20260926 23:23:41.961139 5525 tablet_bootstrap.cc:492] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6: No bootstrap required, opened a new log
I20260926 23:23:41.961390 5523 ts_tablet_manager.cc:1428] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 23:23:41.961400 5525 ts_tablet_manager.cc:1397] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260926 23:23:41.964031 5524 raft_consensus.cc:348] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:41.964483 5524 raft_consensus.cc:374] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:41.964577 5524 raft_consensus.cc:729] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7258cc7838c465586f251f6e05f0431, State: Initialized, Role: FOLLOWER
I20260926 23:23:41.965233 5524 consensus_queue.cc:260] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:41.972500 5375 heartbeater.cc:503] Master 127.0.97.62:33369 was elected leader, sending a full tablet report...
I20260926 23:23:41.973697 5524 ts_tablet_manager.cc:1428] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20260926 23:23:41.973816 5525 raft_consensus.cc:348] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:41.974294 5525 raft_consensus.cc:374] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:41.974412 5525 raft_consensus.cc:729] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6acf6e3cf9c64967b3bd381fa7e46fe6, State: Initialized, Role: FOLLOWER
I20260926 23:23:41.975037 5525 consensus_queue.cc:260] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:41.976864 5525 ts_tablet_manager.cc:1428] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 23:23:42.041635 5531 raft_consensus.cc:482] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:42.042335 5531 raft_consensus.cc:504] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:42.044963 5531 leader_election.cc:288] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dce70ba69db747d6a0e850c855211e4b (127.0.97.3:44365), d7258cc7838c465586f251f6e05f0431 (127.0.97.2:46089)
I20260926 23:23:42.050132 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.047517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45121 (local address 127.0.97.3:44365)
0926 23:23:42.047771 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:42.047775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:42.047793 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:23:42.047846 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:42.047849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:42.047913 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:42.048103 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:42.049494 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:42.049500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:42.049505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:42.049523 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:42.049562 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:42.049574 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:42.049708 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:42.049783 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 23:23:42.049852 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 23:23:42.050334 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.045286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44365 (local address 127.0.97.1:45121)
0926 23:23:42.046079 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.046102 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:42.046239 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:42.048122 (+ 1883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:42.048127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:42.048154 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:42.049268 (+ 1114us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:42.049272 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:42.049290 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:42.049343 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:42.049669 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:42.049673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:42.049679 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:23:42.049831 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 23:23:42.049884 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":702,"threads_started":1}
I20260926 23:23:42.051610 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50446ac32cad4e0aa126cd53cf9726f5" candidate_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce70ba69db747d6a0e850c855211e4b" is_pre_election: true
I20260926 23:23:42.052225 5460 raft_consensus.cc:2454] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6acf6e3cf9c64967b3bd381fa7e46fe6 in term 0.
I20260926 23:23:42.053112 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.045952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54085 (local address 127.0.97.2:46089)
0926 23:23:42.046139 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:42.046148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:42.050795 (+ 4647us) server_negotiation.cc:407] Connection header received
0926 23:23:42.050921 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:42.050927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:42.050996 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:42.051111 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:42.052353 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:42.052359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:42.052363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:42.052385 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:42.052422 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:42.052439 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:42.052541 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:42.052808 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 23:23:42.052866 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:23:42.053128 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.045839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46089 (local address 127.0.97.1:54085)
0926 23:23:42.050696 (+ 4857us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.050710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:42.050818 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:42.051280 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:42.051284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:42.051328 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:42.052089 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:23:42.052092 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:42.052109 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:23:42.052156 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:42.052678 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:42.052704 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:42.052708 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 23:23:42.052806 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 23:23:42.052860 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4776,"threads_started":1}
I20260926 23:23:42.053268 5132 leader_election.cc:302] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6acf6e3cf9c64967b3bd381fa7e46fe6, dce70ba69db747d6a0e850c855211e4b; no voters:
I20260926 23:23:42.053778 5531 raft_consensus.cc:2781] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:42.053939 5531 raft_consensus.cc:482] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:42.054071 5531 raft_consensus.cc:3037] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:42.056546 5531 raft_consensus.cc:504] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:42.057394 5531 leader_election.cc:288] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [CANDIDATE]: Term 1 election: Requested vote from peers dce70ba69db747d6a0e850c855211e4b (127.0.97.3:44365), d7258cc7838c465586f251f6e05f0431 (127.0.97.2:46089)
I20260926 23:23:42.057823 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50446ac32cad4e0aa126cd53cf9726f5" candidate_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce70ba69db747d6a0e850c855211e4b"
I20260926 23:23:42.058032 5460 raft_consensus.cc:3037] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:42.058327 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50446ac32cad4e0aa126cd53cf9726f5" candidate_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7258cc7838c465586f251f6e05f0431" is_pre_election: true
I20260926 23:23:42.058387 5329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50446ac32cad4e0aa126cd53cf9726f5" candidate_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7258cc7838c465586f251f6e05f0431"
I20260926 23:23:42.058905 5330 raft_consensus.cc:2454] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6acf6e3cf9c64967b3bd381fa7e46fe6 in term 0.
I20260926 23:23:42.060639 5460 raft_consensus.cc:2454] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6acf6e3cf9c64967b3bd381fa7e46fe6 in term 1.
I20260926 23:23:42.061125 5132 leader_election.cc:302] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6acf6e3cf9c64967b3bd381fa7e46fe6, dce70ba69db747d6a0e850c855211e4b; no voters: d7258cc7838c465586f251f6e05f0431
I20260926 23:23:42.061435 5531 raft_consensus.cc:2781] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:42.061964 5531 raft_consensus.cc:686] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 1 LEADER]: Becoming Leader. State: Replica: 6acf6e3cf9c64967b3bd381fa7e46fe6, State: Running, Role: LEADER
I20260926 23:23:42.062537 5531 consensus_queue.cc:237] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } }
I20260926 23:23:42.066913 5057 catalog_manager.cc:5495] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6acf6e3cf9c64967b3bd381fa7e46fe6 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6acf6e3cf9c64967b3bd381fa7e46fe6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 } health_report { overall_health: UNKNOWN } } }
I20260926 23:23:42.081043 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.078906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.0.97.1:43401)
0926 23:23:42.079160 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:42.079167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:42.079190 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:23:42.079290 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:42.079294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:42.079372 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:42.079466 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:42.080346 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:42.080351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:42.080354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:42.080372 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:42.080412 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:42.080426 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:42.080530 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:42.080598 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 23:23:42.080648 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:23:42.089254 5329 raft_consensus.cc:3037] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:42.089691 5460 raft_consensus.cc:1261] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 1 FOLLOWER]: Refusing update from remote peer 6acf6e3cf9c64967b3bd381fa7e46fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:42.090720 5534 consensus_queue.cc:1035] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dce70ba69db747d6a0e850c855211e4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:42.092203 5329 raft_consensus.cc:1261] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 1 FOLLOWER]: Refusing update from remote peer 6acf6e3cf9c64967b3bd381fa7e46fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:42.093015 5534 consensus_queue.cc:1035] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7258cc7838c465586f251f6e05f0431" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:23:42.101575 5506 tablet.cc:2369] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:42.103366 5538 mvcc.cc:205] Tried to move back new op lower bound from 7333744730468225024 to 7333744730371760128. Current Snapshot: MvccSnapshot[applied={T|T < 7333744730468225024}]
I20260926 23:23:42.105062 5537 mvcc.cc:205] Tried to move back new op lower bound from 7333744730468225024 to 7333744730371760128. Current Snapshot: MvccSnapshot[applied={T|T < 7333744730468225024}]
I20260926 23:23:42.109839 5540 mvcc.cc:205] Tried to move back new op lower bound from 7333744730468225024 to 7333744730371760128. Current Snapshot: MvccSnapshot[applied={T|T < 7333744730468225024}]
I20260926 23:23:42.118912 5057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:42.119189 5057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:42.122591 5057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29fb3fa8bdfd4e8297aa45140e0e840a: Sending DeleteTablet for 3 replicas of tablet 50446ac32cad4e0aa126cd53cf9726f5
I20260926 23:23:42.123823 5310 tablet_service.cc:1514] Processing DeleteTablet for tablet 50446ac32cad4e0aa126cd53cf9726f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:42 UTC) from {username='slave'} at 127.0.0.1:56398
I20260926 23:23:42.124037 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 50446ac32cad4e0aa126cd53cf9726f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:42 UTC) from {username='slave'} at 127.0.0.1:52900
I20260926 23:23:42.124603 5179 tablet_service.cc:1514] Processing DeleteTablet for tablet 50446ac32cad4e0aa126cd53cf9726f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:42 UTC) from {username='slave'} at 127.0.0.1:39620
I20260926 23:23:42.124727 5546 tablet_replica.cc:330] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: stopping tablet replica
I20260926 23:23:42.125229 5546 raft_consensus.cc:2229] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:42.125804 5546 raft_consensus.cc:2258] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:42.126147 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5117
I20260926 23:23:42.128386 5547 tablet_replica.cc:330] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: stopping tablet replica
I20260926 23:23:42.129045 5547 raft_consensus.cc:2229] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:42.129788 5547 raft_consensus.cc:2258] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:42.132175 5546 ts_tablet_manager.cc:1905] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:42.132797 5547 ts_tablet_manager.cc:1905] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:42.141671 5546 ts_tablet_manager.cc:1918] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:42.141964 5546 log.cc:1196] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/50446ac32cad4e0aa126cd53cf9726f5
I20260926 23:23:42.142540 5546 ts_tablet_manager.cc:1939] T 50446ac32cad4e0aa126cd53cf9726f5 P d7258cc7838c465586f251f6e05f0431: Deleting consensus metadata
I20260926 23:23:42.143992 5547 ts_tablet_manager.cc:1918] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:42.144183 5547 log.cc:1196] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/50446ac32cad4e0aa126cd53cf9726f5
I20260926 23:23:42.144735 5547 ts_tablet_manager.cc:1939] T 50446ac32cad4e0aa126cd53cf9726f5 P dce70ba69db747d6a0e850c855211e4b: Deleting consensus metadata
I20260926 23:23:42.144909 5043 catalog_manager.cc:4841] TS d7258cc7838c465586f251f6e05f0431 (127.0.97.2:46089): tablet 50446ac32cad4e0aa126cd53cf9726f5 (table test-table [id=e61bb02a25c1485fbc442a3f1a0e480d]) successfully deleted
I20260926 23:23:42.145231 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.127291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.0.97.1:43401)
0926 23:23:42.127465 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:42.127474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:42.140818 (+ 13344us) server_negotiation.cc:407] Connection header received
0926 23:23:42.141943 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:42.141949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:42.142024 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:42.142110 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:42.144181 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:42.144188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:42.144193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:42.144217 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:42.144259 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:42.144276 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:42.144387 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:42.144837 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 23:23:42.144908 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:23:42.146023 5548 tablet_replica.cc:330] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6: stopping tablet replica
I20260926 23:23:42.146449 5239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:42.146727 5042 catalog_manager.cc:4841] TS dce70ba69db747d6a0e850c855211e4b (127.0.97.3:44365): tablet 50446ac32cad4e0aa126cd53cf9726f5 (table test-table [id=e61bb02a25c1485fbc442a3f1a0e480d]) successfully deleted
I20260926 23:23:42.146672 5548 raft_consensus.cc:2229] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:42.241959 5548 raft_consensus.cc:2258] T 50446ac32cad4e0aa126cd53cf9726f5 P 6acf6e3cf9c64967b3bd381fa7e46fe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:42.242069 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5117
W20260926 23:23:42.261510 5042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43401 (error 108)
I20260926 23:23:42.261631 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5248
I20260926 23:23:42.263499 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.263023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39646)
0926 23:23:42.263197 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.263332 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:23:42.263824 5042 catalog_manager.cc:4584] TS 6acf6e3cf9c64967b3bd381fa7e46fe6 (127.0.97.1:43401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50446ac32cad4e0aa126cd53cf9726f5: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
I20260926 23:23:42.264770 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.262532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.0.97.2:46089)
0926 23:23:42.262736 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:42.262742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:42.262765 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:23:42.262842 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:42.262847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:42.262959 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:42.263069 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:42.264121 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:42.264126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:42.264129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:42.264146 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:42.264178 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:42.264191 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:42.264308 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:42.264417 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 23:23:42.264467 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 23:23:42.265519 5370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:42.318266 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.317563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39656)
0926 23:23:42.317818 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.318008 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 23:23:42.319808 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.319311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39662)
0926 23:23:42.319516 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.319611 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 23:23:42.360266 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5248
I20260926 23:23:42.376915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5378
I20260926 23:23:42.379745 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.377821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.0.97.3:44365)
0926 23:23:42.378001 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:42.378006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:42.378025 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:23:42.378144 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:42.378148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:42.378223 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:42.378308 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:42.379139 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:42.379143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:42.379146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:42.379160 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:42.379191 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:42.379202 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:42.379307 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:42.379377 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 23:23:42.379422 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:23:42.380525 5500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:42.380839 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.380334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39672)
0926 23:23:42.380490 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.380576 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 23:23:42.382165 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.381623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39680)
0926 23:23:42.381876 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.381977 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 23:23:42.462929 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.462435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39692)
0926 23:23:42.462682 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.462779 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 23:23:42.464193 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:42.463739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43401 (local address 127.0.0.1:39696)
0926 23:23:42.463982 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:42.464070 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 23:23:42.468448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5378
I20260926 23:23:42.484932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5027
I20260926 23:23:42.486104 5088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:42.590488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5027
2026-09-26T23:23:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2772 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 23:23:42.631757 388 mini_oidc.cc:152] Starting JWKS server
I20260926 23:23:42.634413 388 webserver.cc:460] Webserver started at https://127.0.0.1:44539/ using document root <none> and password file <none>
I20260926 23:23:42.634742 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:23:42.635399 388 webserver.cc:460] Webserver started at http://127.0.0.1:43099/ using document root <none> and password file <none>
2026-09-26T23:23:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:42Z Disabled control of system clock
I20260926 23:23:42.662474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42767
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42767
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44539/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/testchainca.cert with env {}
W20260926 23:23:42.985850 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:42.986239 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:42.986400 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:42.993897 5560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:42.994093 5560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:42.994189 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:42.994299 5560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:42.994375 5560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:23:42.994465 5560 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:23:42.994542 5560 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44539/jwks.json
I20260926 23:23:43.005515 5560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44539/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:43.006891 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:43.008262 5560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:43.021070 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:43.021101 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:43.021139 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:43.021929 5560 server_base.cc:1034] running on GCE node
I20260926 23:23:43.022746 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:43.024170 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:43.025374 5560 hybrid_clock.cc:648] HybridClock initialized: now 1790465023025323 us; error 41 us; skew 500 ppm
I20260926 23:23:43.025756 5560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:43.028126 5560 webserver.cc:460] Webserver started at http://127.0.97.62:41547/ using document root <none> and password file <none>
I20260926 23:23:43.028928 5560 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:43.029050 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:43.029388 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:43.032388 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "c644147daae54183a570b1cab5f28579"
format_stamp: "Formatted at 2026-09-26 23:23:43 on dist-test-slave-q5g8"
server_key: "bd6e996def82e80f39061cdaef8ce6c1"
server_key_iv: "3516cb397f5f6df0a6dbcb6828438729"
server_key_version: "encryptionkey@0"
I20260926 23:23:43.033315 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "c644147daae54183a570b1cab5f28579"
format_stamp: "Formatted at 2026-09-26 23:23:43 on dist-test-slave-q5g8"
server_key: "bd6e996def82e80f39061cdaef8ce6c1"
server_key_iv: "3516cb397f5f6df0a6dbcb6828438729"
server_key_version: "encryptionkey@0"
I20260926 23:23:43.038748 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 23:23:43.042421 5574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:43.043917 5560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:23:43.044118 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "c644147daae54183a570b1cab5f28579"
format_stamp: "Formatted at 2026-09-26 23:23:43 on dist-test-slave-q5g8"
server_key: "bd6e996def82e80f39061cdaef8ce6c1"
server_key_iv: "3516cb397f5f6df0a6dbcb6828438729"
server_key_version: "encryptionkey@0"
I20260926 23:23:43.044348 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:43.061058 5560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:43.061832 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:43.062062 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:43.080044 5560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42767
I20260926 23:23:43.080062 5625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42767 every 8 connection(s)
I20260926 23:23:43.081579 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:43.085527 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:43.091912 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5560
I20260926 23:23:43.092119 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:43.092942 388 external_mini_cluster.cc:1428] Setting key 9744b347c5a8c225132c36f0c5a6cceb
I20260926 23:23:43.097460 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579: Bootstrap starting.
I20260926 23:23:43.101094 5626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:43.102392 5626 log.cc:824] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:43.105917 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579: No bootstrap required, opened a new log
I20260926 23:23:43.108414 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:43.095167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.0.97.62:42767)
0926 23:23:43.096269 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:43.096281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:43.096325 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:23:43.098203 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:43.098227 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:43.098726 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:43.099094 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:43.099818 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.101426 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.103287 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.104201 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.104339 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:43.105738 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:43.105746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:43.105760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:43.106429 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:43.106513 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:43.106536 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:43.106635 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:43.106930 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 23:23:43.107306 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":424,"threads_started":1}
I20260926 23:23:43.112919 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:23:43.115868 5626 raft_consensus.cc:348] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c644147daae54183a570b1cab5f28579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42767 } }
I20260926 23:23:43.116293 5626 raft_consensus.cc:374] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:43.116412 5626 raft_consensus.cc:729] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c644147daae54183a570b1cab5f28579, State: Initialized, Role: FOLLOWER
I20260926 23:23:43.117046 5626 consensus_queue.cc:260] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c644147daae54183a570b1cab5f28579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42767 } }
I20260926 23:23:43.117233 5626 raft_consensus.cc:388] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:43.117327 5626 raft_consensus.cc:482] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:43.117448 5626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:43.119486 5626 raft_consensus.cc:504] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c644147daae54183a570b1cab5f28579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42767 } }
I20260926 23:23:43.119922 5626 leader_election.cc:302] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c644147daae54183a570b1cab5f28579; no voters:
I20260926 23:23:43.120440 5626 leader_election.cc:288] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:43.120636 5631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:43.122169 5631 raft_consensus.cc:686] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [term 1 LEADER]: Becoming Leader. State: Replica: c644147daae54183a570b1cab5f28579, State: Running, Role: LEADER
I20260926 23:23:43.122781 5626 sys_catalog.cc:564] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:43.122748 5631 consensus_queue.cc:237] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c644147daae54183a570b1cab5f28579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42767 } }
I20260926 23:23:43.126063 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c644147daae54183a570b1cab5f28579. Latest consensus state: current_term: 1 leader_uuid: "c644147daae54183a570b1cab5f28579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c644147daae54183a570b1cab5f28579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42767 } } }
I20260926 23:23:43.126111 5632 sys_catalog.cc:455] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c644147daae54183a570b1cab5f28579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c644147daae54183a570b1cab5f28579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42767 } } }
I20260926 23:23:43.126734 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:43.126814 5632 sys_catalog.cc:458] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:43.128860 5639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:43.134125 5639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:43.143452 5639 catalog_manager.cc:1347] Generated new cluster ID: a60054fedb9d4d24b76c52f0b0c25ce6
I20260926 23:23:43.143584 5639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:43.162899 5639 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:43.164800 5639 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:43.175045 5639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c644147daae54183a570b1cab5f28579: Generated new TSK 0
I20260926 23:23:43.175980 5639 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 23:23:43.218284 5656 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 23:23:43.218464 5656 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:23:43.219691 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:43.197168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.97.62:42767)
0926 23:23:43.197386 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:43.197393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:43.197935 (+ 542us) server_negotiation.cc:407] Connection header received
0926 23:23:43.198246 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:43.198252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:43.198351 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:43.198516 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:23:43.199072 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.200198 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.201699 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.202399 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.202533 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:43.203056 (+ 523us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:23:43.218991 (+ 15935us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 23:23:43.219376 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 23:23:43.219828 5629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 23:23:43.221264 5655 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:43.197070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42767 (local address 127.0.0.1:42786)
0926 23:23:43.197817 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:43.197836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:43.198000 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:43.198580 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:43.198584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:43.198594 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:23:43.198919 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:43.198927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.200341 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.200344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:43.201554 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:43.201567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.202519 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.202574 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:43.202698 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:43.202755 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:23:43.219329 (+ 16574us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 23:23:43.219620 (+ 291us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":380,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 23:23:43.223389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5560
I20260926 23:23:43.224640 5621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:43.320060 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5560
2026-09-26T23:23:43Z chronyd exiting
I20260926 23:23:43.367573 388 test_util.cc:175] -----------------------------------------------
I20260926 23:23:43.367743 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790464978839002-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 23:23:43.371990 388 mini_oidc.cc:152] Starting JWKS server
I20260926 23:23:43.374159 388 webserver.cc:460] Webserver started at https://127.0.0.1:46273/ using document root <none> and password file <none>
I20260926 23:23:43.374457 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:23:43.375159 388 webserver.cc:460] Webserver started at http://127.0.0.1:36253/ using document root <none> and password file <none>
2026-09-26T23:23:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:43Z Disabled control of system clock
I20260926 23:23:43.403183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41877
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41877
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46273/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/testchainca.cert with env {}
W20260926 23:23:43.745831 5663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:43.746232 5663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:43.746419 5663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:43.753871 5663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:43.754053 5663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:43.754163 5663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:43.754292 5663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:43.754388 5663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:23:43.754488 5663 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:23:43.754575 5663 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46273/jwks.json
I20260926 23:23:43.765127 5663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46273/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:43.766613 5663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:43.768009 5663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:43.781194 5668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:43.781217 5671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:43.781147 5669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:43.782141 5663 server_base.cc:1034] running on GCE node
I20260926 23:23:43.782979 5663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:43.784473 5663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:43.785706 5663 hybrid_clock.cc:648] HybridClock initialized: now 1790465023785639 us; error 65 us; skew 500 ppm
I20260926 23:23:43.786099 5663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:43.792524 5663 webserver.cc:460] Webserver started at http://127.0.97.62:39329/ using document root <none> and password file <none>
I20260926 23:23:43.793351 5663 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:43.793524 5663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:43.793885 5663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:43.797008 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "00a3806c15f845efa85f572320fdf5ad"
format_stamp: "Formatted at 2026-09-26 23:23:43 on dist-test-slave-q5g8"
server_key: "0aebd87224ff9dfc0670d31a1a7bbe62"
server_key_iv: "b85e12f47ac45fa2d08626fc7aa414ec"
server_key_version: "encryptionkey@0"
I20260926 23:23:43.797855 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "00a3806c15f845efa85f572320fdf5ad"
format_stamp: "Formatted at 2026-09-26 23:23:43 on dist-test-slave-q5g8"
server_key: "0aebd87224ff9dfc0670d31a1a7bbe62"
server_key_iv: "b85e12f47ac45fa2d08626fc7aa414ec"
server_key_version: "encryptionkey@0"
I20260926 23:23:43.803295 5663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:43.806859 5677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:43.808390 5663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:23:43.808609 5663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "00a3806c15f845efa85f572320fdf5ad"
format_stamp: "Formatted at 2026-09-26 23:23:43 on dist-test-slave-q5g8"
server_key: "0aebd87224ff9dfc0670d31a1a7bbe62"
server_key_iv: "b85e12f47ac45fa2d08626fc7aa414ec"
server_key_version: "encryptionkey@0"
I20260926 23:23:43.808907 5663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:43.830837 5663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:43.831722 5663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:43.832029 5663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:43.850559 5663 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41877
I20260926 23:23:43.850584 5728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41877 every 8 connection(s)
I20260926 23:23:43.852067 5663 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:43.854385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5663
I20260926 23:23:43.854620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:43.855429 388 external_mini_cluster.cc:1428] Setting key 20c1f2580ed5b7d62c5af93030519448
I20260926 23:23:43.857163 5729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:43.870293 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:43.857591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.0.97.62:41877)
0926 23:23:43.858700 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:43.858713 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:43.858764 (+ 51us) server_negotiation.cc:407] Connection header received
0926 23:23:43.860456 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:43.860505 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:43.861182 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:43.861613 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:43.862238 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.863677 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.865582 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.866285 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.866456 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:43.867728 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:43.867736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:43.867756 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:43.868399 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:43.868465 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:43.868488 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:43.868584 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:43.868876 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 23:23:43.869294 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":425,"threads_started":1}
I20260926 23:23:43.871341 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad: Bootstrap starting.
I20260926 23:23:43.874908 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:23:43.875283 5729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:43.876399 5729 log.cc:824] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:43.879670 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad: No bootstrap required, opened a new log
I20260926 23:23:43.887032 5729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a3806c15f845efa85f572320fdf5ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260926 23:23:43.887430 5729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:43.887535 5729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3806c15f845efa85f572320fdf5ad, State: Initialized, Role: FOLLOWER
I20260926 23:23:43.888202 5729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a3806c15f845efa85f572320fdf5ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260926 23:23:43.888443 5729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:43.888586 5729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:43.888789 5729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:43.891014 5729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a3806c15f845efa85f572320fdf5ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260926 23:23:43.891573 5729 leader_election.cc:302] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a3806c15f845efa85f572320fdf5ad; no voters:
I20260926 23:23:43.892228 5729 leader_election.cc:288] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:43.892408 5734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:43.893168 5734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [term 1 LEADER]: Becoming Leader. State: Replica: 00a3806c15f845efa85f572320fdf5ad, State: Running, Role: LEADER
I20260926 23:23:43.893805 5734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a3806c15f845efa85f572320fdf5ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } }
I20260926 23:23:43.894569 5729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:43.896467 5735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a3806c15f845efa85f572320fdf5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a3806c15f845efa85f572320fdf5ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } } }
I20260926 23:23:43.896858 5735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:43.897687 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a3806c15f845efa85f572320fdf5ad. Latest consensus state: current_term: 1 leader_uuid: "00a3806c15f845efa85f572320fdf5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a3806c15f845efa85f572320fdf5ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41877 } } }
I20260926 23:23:43.898017 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:43.898715 5739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:43.906256 5739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:43.915158 5739 catalog_manager.cc:1347] Generated new cluster ID: 0fbd82c4bb9046d3a2d5b452dd92f7eb
I20260926 23:23:43.915290 5739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:43.925299 5739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:43.926612 5739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:43.933210 5739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a3806c15f845efa85f572320fdf5ad: Generated new TSK 0
I20260926 23:23:43.934134 5739 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 23:23:43.977761 5759 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 23:23:43.977905 5759 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:23:43.979053 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:43.951977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.0.97.62:41877)
0926 23:23:43.952202 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:43.952210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:43.952559 (+ 349us) server_negotiation.cc:407] Connection header received
0926 23:23:43.952838 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:43.952842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:43.952907 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:43.953076 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:23:43.953557 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.954282 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.955647 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.956120 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.956255 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:43.956799 (+ 544us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:23:43.978312 (+ 21513us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 23:23:43.978720 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 23:23:43.979187 5732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 23:23:43.979293 5758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:43.951861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41877 (local address 127.0.0.1:41250)
0926 23:23:43.952454 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:43.952468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:43.952634 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:43.953087 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:43.953092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:43.953104 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:23:43.953392 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:43.953402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.954428 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.954435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:43.955504 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:43.955514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:43.956242 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:43.956297 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:43.956458 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:43.956491 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:23:43.978587 (+ 22096us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 23:23:43.978876 (+ 289us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":322,"threads_started":1}
I20260926 23:23:43.980450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5663
I20260926 23:23:43.981822 5724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:44.078480 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5663
2026-09-26T23:23:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (764 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 23:23:44.136348 388 mini_oidc.cc:152] Starting JWKS server
I20260926 23:23:44.138350 388 webserver.cc:460] Webserver started at https://127.0.0.1:46501/ using document root <none> and password file <none>
I20260926 23:23:44.138609 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:23:44.139312 388 webserver.cc:460] Webserver started at http://127.0.0.1:42955/ using document root <none> and password file <none>
2026-09-26T23:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:44Z Disabled control of system clock
I20260926 23:23:44.166867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32881
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 23:23:44.505162 5766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:44.505465 5766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:44.505545 5766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:44.512630 5766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:44.512816 5766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:44.512884 5766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:44.512946 5766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:44.512989 5766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:23:44.513051 5766 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:23:44.513090 5766 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46501/jwks.json
I20260926 23:23:44.522281 5766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:44.523582 5766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:44.525301 5766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:44.537166 5771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:44.537168 5772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:44.537190 5774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:44.538177 5766 server_base.cc:1034] running on GCE node
I20260926 23:23:44.539083 5766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:44.540565 5766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:44.541796 5766 hybrid_clock.cc:648] HybridClock initialized: now 1790465024541746 us; error 42 us; skew 500 ppm
I20260926 23:23:44.542140 5766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:44.544467 5766 webserver.cc:460] Webserver started at http://127.0.97.62:34631/ using document root <none> and password file <none>
I20260926 23:23:44.545220 5766 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:44.545310 5766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:44.545574 5766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:44.548367 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "53bb01b129fe43a6a721e762b24005a7"
format_stamp: "Formatted at 2026-09-26 23:23:44 on dist-test-slave-q5g8"
server_key: "a555cf78f3507d983986d28455c5f073"
server_key_iv: "297ef5eb6e26021e9938508792ec4130"
server_key_version: "encryptionkey@0"
I20260926 23:23:44.549176 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "53bb01b129fe43a6a721e762b24005a7"
format_stamp: "Formatted at 2026-09-26 23:23:44 on dist-test-slave-q5g8"
server_key: "a555cf78f3507d983986d28455c5f073"
server_key_iv: "297ef5eb6e26021e9938508792ec4130"
server_key_version: "encryptionkey@0"
I20260926 23:23:44.554301 5766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:23:44.557868 5780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:44.559268 5766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:44.559450 5766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "53bb01b129fe43a6a721e762b24005a7"
format_stamp: "Formatted at 2026-09-26 23:23:44 on dist-test-slave-q5g8"
server_key: "a555cf78f3507d983986d28455c5f073"
server_key_iv: "297ef5eb6e26021e9938508792ec4130"
server_key_version: "encryptionkey@0"
I20260926 23:23:44.559671 5766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:44.576604 5766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:44.577551 5766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:44.577857 5766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:44.597020 5766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32881
I20260926 23:23:44.597034 5831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32881 every 8 connection(s)
I20260926 23:23:44.598389 5766 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:44.602098 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:44.608639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5766
I20260926 23:23:44.608875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:44.609678 388 external_mini_cluster.cc:1428] Setting key 8f7fe552d97a57b213acf8ae7fefda59
I20260926 23:23:44.613647 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7: Bootstrap starting.
I20260926 23:23:44.616902 5832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:44.617952 5832 log.cc:824] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:44.620867 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7: No bootstrap required, opened a new log
I20260926 23:23:44.624119 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:44.611814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.0.97.62:32881)
0926 23:23:44.612907 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:44.612927 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:44.612971 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:23:44.614751 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:44.614774 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:44.615278 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:44.615676 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:44.616244 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:44.617644 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:44.619010 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:44.619694 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:44.619901 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:44.621122 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:44.621133 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:44.621155 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:44.621915 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:44.622016 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:44.622044 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:44.622185 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:44.622393 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 23:23:44.622736 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":328,"threads_started":1}
I20260926 23:23:44.628113 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:23:44.630273 5832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bb01b129fe43a6a721e762b24005a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32881 } }
I20260926 23:23:44.630720 5832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:44.630831 5832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bb01b129fe43a6a721e762b24005a7, State: Initialized, Role: FOLLOWER
I20260926 23:23:44.631469 5832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bb01b129fe43a6a721e762b24005a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32881 } }
I20260926 23:23:44.631721 5832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:44.631872 5832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:44.632047 5832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:44.634322 5832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bb01b129fe43a6a721e762b24005a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32881 } }
I20260926 23:23:44.634825 5832 leader_election.cc:302] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53bb01b129fe43a6a721e762b24005a7; no voters:
I20260926 23:23:44.635535 5832 leader_election.cc:288] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:44.635700 5837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:44.636444 5837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [term 1 LEADER]: Becoming Leader. State: Replica: 53bb01b129fe43a6a721e762b24005a7, State: Running, Role: LEADER
I20260926 23:23:44.637013 5837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bb01b129fe43a6a721e762b24005a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32881 } }
I20260926 23:23:44.637687 5832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:44.639989 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53bb01b129fe43a6a721e762b24005a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bb01b129fe43a6a721e762b24005a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32881 } } }
I20260926 23:23:44.640414 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:44.641812 5844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:44.641716 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53bb01b129fe43a6a721e762b24005a7. Latest consensus state: current_term: 1 leader_uuid: "53bb01b129fe43a6a721e762b24005a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bb01b129fe43a6a721e762b24005a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32881 } } }
I20260926 23:23:44.642027 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:44.647953 5844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:44.659503 5844 catalog_manager.cc:1347] Generated new cluster ID: 54ec5cc0e0a84599bfa09a0daeef1ecc
I20260926 23:23:44.659639 5844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:44.672478 5844 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:44.674369 5844 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:44.684696 5844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53bb01b129fe43a6a721e762b24005a7: Generated new TSK 0
I20260926 23:23:44.685758 5844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:23:44.721156 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:44.712719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.0.97.62:32881)
0926 23:23:44.712959 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:44.712966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:44.713358 (+ 392us) server_negotiation.cc:407] Connection header received
0926 23:23:44.713576 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:44.713580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:44.713651 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:44.713834 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:23:44.714404 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:44.715182 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:44.717244 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:44.717789 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:44.717904 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:44.718487 (+ 583us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:23:44.720448 (+ 1961us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 23:23:44.720822 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 23:23:44.721309 5861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:44.712555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32881 (local address 127.0.0.1:51946)
0926 23:23:44.713240 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:44.713266 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:44.713420 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:44.713872 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:44.713878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:44.713892 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:23:44.714235 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:44.714246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:44.715338 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:44.715343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:44.717078 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:44.717092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:44.717921 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:44.717994 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:44.718174 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:44.718220 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:23:44.720638 (+ 2418us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 23:23:44.720904 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:32881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":375,"threads_started":1}
W20260926 23:23:44.721624 5835 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 23:23:44.722419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5766
I20260926 23:23:44.723524 5827 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 23:23:44.733265 5862 webserver.cc:533] Webserver: sslize failed:
E20260926 23:23:44.733376 5862 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:23:44.815107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5766
2026-09-26T23:23:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (734 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T23:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:44Z Disabled control of system clock
I20260926 23:23:44.897400 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38979
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38979
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/testchainca.cert with env {}
W20260926 23:23:45.220114 5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:45.220450 5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:45.220538 5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:45.227898 5867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:45.228008 5867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:45.228046 5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:45.228094 5867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:45.228138 5867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:23:45.228197 5867 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:23:45.228236 5867 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 23:23:45.238286 5867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:45.239547 5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:45.240788 5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:45.252899 5873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:45.252946 5872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:45.252880 5875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:45.253475 5867 server_base.cc:1034] running on GCE node
I20260926 23:23:45.254483 5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:45.256047 5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:45.257241 5867 hybrid_clock.cc:648] HybridClock initialized: now 1790465025257177 us; error 56 us; skew 500 ppm
I20260926 23:23:45.257591 5867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:23:45.259935 5867 webserver.cc:460] Webserver started at http://127.0.97.62:42111/ using document root <none> and password file <none>
I20260926 23:23:45.260648 5867 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:45.260842 5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:45.261209 5867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:45.264899 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "5fd13de0623844d1bfffe9c6866c5020"
format_stamp: "Formatted at 2026-09-26 23:23:45 on dist-test-slave-q5g8"
server_key: "ac814a5ea8c36d055168fc23b8d69b08"
server_key_iv: "23205bb4627083e8fb66dae6db2a2ed1"
server_key_version: "encryptionkey@0"
I20260926 23:23:45.265700 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "5fd13de0623844d1bfffe9c6866c5020"
format_stamp: "Formatted at 2026-09-26 23:23:45 on dist-test-slave-q5g8"
server_key: "ac814a5ea8c36d055168fc23b8d69b08"
server_key_iv: "23205bb4627083e8fb66dae6db2a2ed1"
server_key_version: "encryptionkey@0"
I20260926 23:23:45.270789 5867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 23:23:45.274296 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:45.275660 5867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 23:23:45.275853 5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "5fd13de0623844d1bfffe9c6866c5020"
format_stamp: "Formatted at 2026-09-26 23:23:45 on dist-test-slave-q5g8"
server_key: "ac814a5ea8c36d055168fc23b8d69b08"
server_key_iv: "23205bb4627083e8fb66dae6db2a2ed1"
server_key_version: "encryptionkey@0"
I20260926 23:23:45.276075 5867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:45.296356 5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:45.297338 5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:45.297639 5867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:45.316072 5867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38979
I20260926 23:23:45.316093 5932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38979 every 8 connection(s)
I20260926 23:23:45.317567 5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:45.321477 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:45.327036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5867
I20260926 23:23:45.327265 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:45.328078 388 external_mini_cluster.cc:1428] Setting key 86ab607482e9472f7b42d60992fcb122
I20260926 23:23:45.332947 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020: Bootstrap starting.
I20260926 23:23:45.336090 5933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:45.337159 5933 log.cc:824] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:45.340569 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020: No bootstrap required, opened a new log
I20260926 23:23:45.342331 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:45.330204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.0.97.62:38979)
0926 23:23:45.331280 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:45.331292 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:45.331336 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:23:45.333171 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:45.333197 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:45.333690 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:45.334117 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:45.334713 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:45.335966 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:45.337375 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:45.338526 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:45.338697 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:45.339968 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:45.339975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:45.339989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:23:45.340536 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:45.340644 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:23:45.340694 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:45.340837 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:45.341063 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 23:23:45.341407 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":370,"threads_started":1}
I20260926 23:23:45.346222 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:23:45.349545 5933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd13de0623844d1bfffe9c6866c5020" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38979 } }
I20260926 23:23:45.349927 5933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:45.350029 5933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd13de0623844d1bfffe9c6866c5020, State: Initialized, Role: FOLLOWER
I20260926 23:23:45.350618 5933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd13de0623844d1bfffe9c6866c5020" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38979 } }
I20260926 23:23:45.350852 5933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:45.350998 5933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:45.351171 5933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:45.353423 5933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd13de0623844d1bfffe9c6866c5020" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38979 } }
I20260926 23:23:45.353948 5933 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd13de0623844d1bfffe9c6866c5020; no voters:
I20260926 23:23:45.354557 5933 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:45.354818 5938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:45.355506 5938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [term 1 LEADER]: Becoming Leader. State: Replica: 5fd13de0623844d1bfffe9c6866c5020, State: Running, Role: LEADER
I20260926 23:23:45.356072 5938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd13de0623844d1bfffe9c6866c5020" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38979 } }
I20260926 23:23:45.356988 5933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:45.359180 5939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd13de0623844d1bfffe9c6866c5020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd13de0623844d1bfffe9c6866c5020" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38979 } } }
I20260926 23:23:45.359154 5940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd13de0623844d1bfffe9c6866c5020. Latest consensus state: current_term: 1 leader_uuid: "5fd13de0623844d1bfffe9c6866c5020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd13de0623844d1bfffe9c6866c5020" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38979 } } }
I20260926 23:23:45.359661 5940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:45.359661 5939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:45.361852 5944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:45.367229 5944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:45.376966 5944 catalog_manager.cc:1347] Generated new cluster ID: 5dde7ba202104bd585b79fe5a90605ad
I20260926 23:23:45.377120 5944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:45.393937 5944 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:45.395862 5944 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:45.406525 5944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd13de0623844d1bfffe9c6866c5020: Generated new TSK 0
I20260926 23:23:45.407407 5944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:23:45.496994 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:45.426501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.97.62:38979)
0926 23:23:45.426751 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:45.426761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:45.427233 (+ 472us) server_negotiation.cc:407] Connection header received
0926 23:23:45.427507 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:45.427512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:45.427583 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:45.427735 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:23:45.428276 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:45.429197 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:45.431264 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:45.431766 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:45.431880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:45.432469 (+ 589us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:23:45.496320 (+ 63851us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 23:23:45.496723 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37210: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 23:23:45.497166 5936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37210: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 23:23:45.497406 5962 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:45.426450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38979 (local address 127.0.0.1:37210)
0926 23:23:45.427094 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:45.427116 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:45.427287 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:45.427772 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:45.427778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:45.427793 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:23:45.428095 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:45.428107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:45.429351 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:45.429357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:45.431091 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:45.431110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:45.431893 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:45.431949 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:45.432141 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:45.432189 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:23:45.496625 (+ 64436us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 23:23:45.496967 (+ 342us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38979: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":328,"threads_started":1}
I20260926 23:23:45.498729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5867
I20260926 23:23:45.499984 5928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:45.598650 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5867
2026-09-26T23:23:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (762 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (411 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1503 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:47 dist-test-slave-q5g8 krb5kdc[5978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:47 dist-test-slave-q5g8 krb5kdc[5978](info): set up 2 sockets
Sep 26 23:23:47 dist-test-slave-q5g8 krb5kdc[5978](info): commencing operation
krb5kdc: starting...
W20260926 23:23:48.617532 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:48 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:23:48.824393 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46881
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:49.145016 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:49.145354 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:49.145529 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:49.152606 5991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:49.152779 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:49.152882 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:49.152987 5991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:49.153074 5991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:49.162590 5991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:49.163899 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:49.165235 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:49.177033 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:49.177109 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:49.177130 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:49.178002 5991 server_base.cc:1034] running on GCE node
I20260926 23:23:49.178836 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:49.180277 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:49.181511 5991 hybrid_clock.cc:648] HybridClock initialized: now 1790465029181443 us; error 59 us; skew 500 ppm
I20260926 23:23:49.181852 5991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:49 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:49.188347 5991 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:23:49.189859 5991 webserver.cc:460] Webserver started at http://127.0.97.62:32861/ using document root <none> and password file <none>
I20260926 23:23:49.190564 5991 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:49.190708 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:49.191025 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:49.194145 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc"
format_stamp: "Formatted at 2026-09-26 23:23:49 on dist-test-slave-q5g8"
server_key: "a9cd671f685b296b6288ca67f6f345ac"
server_key_iv: "c25f348744dedfc114d076d871fd22bd"
server_key_version: "encryptionkey@0"
I20260926 23:23:49.194901 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc"
format_stamp: "Formatted at 2026-09-26 23:23:49 on dist-test-slave-q5g8"
server_key: "a9cd671f685b296b6288ca67f6f345ac"
server_key_iv: "c25f348744dedfc114d076d871fd22bd"
server_key_version: "encryptionkey@0"
I20260926 23:23:49.200089 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:23:49.203491 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:49.204870 5991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 23:23:49.205049 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc"
format_stamp: "Formatted at 2026-09-26 23:23:49 on dist-test-slave-q5g8"
server_key: "a9cd671f685b296b6288ca67f6f345ac"
server_key_iv: "c25f348744dedfc114d076d871fd22bd"
server_key_version: "encryptionkey@0"
I20260926 23:23:49.205235 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:49.234222 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:49.238576 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:49.238865 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:49.257357 5991 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46881
I20260926 23:23:49.257442 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46881 every 8 connection(s)
I20260926 23:23:49.258690 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:49.262450 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:49.263888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 5991
I20260926 23:23:49.264135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:49.265060 388 external_mini_cluster.cc:1428] Setting key 83e74d354271034148a2e04ddcd96f86
I20260926 23:23:49.276386 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc: Bootstrap starting.
I20260926 23:23:49.280735 6058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:49.281914 6058 log.cc:824] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:23:49 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:49.285101 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc: No bootstrap required, opened a new log
I20260926 23:23:49.290428 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:49.267501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.0.97.62:46881)
0926 23:23:49.268500 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:49.268516 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:49.268920 (+ 404us) server_negotiation.cc:407] Connection header received
0926 23:23:49.271376 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:49.271404 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:49.271923 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:49.272413 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:49.273679 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:49.275683 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:49.277225 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:49.278284 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:49.278475 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:49.283389 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:49.283399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:49.283415 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:49.283450 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:49.286448 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:49.287257 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:49.287263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:49.287269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:49.287426 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:49.287837 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:49.287840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:49.287842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:49.288377 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:49.288576 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:49.289097 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 23:23:49.289421 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":413,"threads_started":1}
I20260926 23:23:49.294348 6058 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46881 } }
I20260926 23:23:49.294768 6058 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:49.294888 6058 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43ac3b4921f4fae9fd08bbaa23da7fc, State: Initialized, Role: FOLLOWER
I20260926 23:23:49.295485 6058 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46881 } }
I20260926 23:23:49.295733 6058 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:49.295887 6058 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:49.296087 6058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:49.298475 6058 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46881 } }
I20260926 23:23:49.298974 6058 leader_election.cc:302] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43ac3b4921f4fae9fd08bbaa23da7fc; no voters:
I20260926 23:23:49.299629 6058 leader_election.cc:288] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:49.299767 6063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:49.300485 6063 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [term 1 LEADER]: Becoming Leader. State: Replica: c43ac3b4921f4fae9fd08bbaa23da7fc, State: Running, Role: LEADER
I20260926 23:23:49.301033 6063 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46881 } }
I20260926 23:23:49.301875 6058 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:49.303884 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46881 } } }
I20260926 23:23:49.304453 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:49.304872 6065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43ac3b4921f4fae9fd08bbaa23da7fc. Latest consensus state: current_term: 1 leader_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac3b4921f4fae9fd08bbaa23da7fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46881 } } }
I20260926 23:23:49.305189 6065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc [sys.catalog]: This master's current role is: LEADER
W20260926 23:23:49.312526 6079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:23:49.312660 6079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:23:49.312831 6072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:49.314432 6072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:49.321820 6072 catalog_manager.cc:1347] Generated new cluster ID: c4914da1c1404f748206ee0f59909d4b
I20260926 23:23:49.321938 6072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:49.343099 6072 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:49.344413 6072 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:49.351836 6072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43ac3b4921f4fae9fd08bbaa23da7fc: Generated new TSK 0
I20260926 23:23:49.352602 6072 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:23:49.400542 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46881
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:49.717515 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:49.717761 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:49.717862 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:49.724726 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:49.724857 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:49.724975 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:49.734241 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:49.735466 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:49.736733 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:49.749847 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:49.749920 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:49.750025 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:49.750546 6084 server_base.cc:1034] running on GCE node
I20260926 23:23:49.751071 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:49.751961 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:49.753152 6084 hybrid_clock.cc:648] HybridClock initialized: now 1790465029753111 us; error 33 us; skew 500 ppm
I20260926 23:23:49.753428 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:49 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:49.756443 6084 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:23:49.757930 6084 webserver.cc:460] Webserver started at http://127.0.97.1:33449/ using document root <none> and password file <none>
I20260926 23:23:49.758577 6084 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:49.758711 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:49.759013 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:49.761981 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "bea18957f9b64311ae5be614505443bd"
format_stamp: "Formatted at 2026-09-26 23:23:49 on dist-test-slave-q5g8"
server_key: "fb60754fb4630ac891533c2adf6199c3"
server_key_iv: "c5974d52edc670ed3b2138c866966de3"
server_key_version: "encryptionkey@0"
I20260926 23:23:49.762728 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bea18957f9b64311ae5be614505443bd"
format_stamp: "Formatted at 2026-09-26 23:23:49 on dist-test-slave-q5g8"
server_key: "fb60754fb4630ac891533c2adf6199c3"
server_key_iv: "c5974d52edc670ed3b2138c866966de3"
server_key_version: "encryptionkey@0"
I20260926 23:23:49.767536 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 23:23:49.770852 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:49.772351 6084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:23:49.772516 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "bea18957f9b64311ae5be614505443bd"
format_stamp: "Formatted at 2026-09-26 23:23:49 on dist-test-slave-q5g8"
server_key: "fb60754fb4630ac891533c2adf6199c3"
server_key_iv: "c5974d52edc670ed3b2138c866966de3"
server_key_version: "encryptionkey@0"
I20260926 23:23:49.772753 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:49.819741 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:49.823964 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:49.824267 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:49.825273 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:49.826552 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:49.826663 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:49.826820 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:49.826895 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:49.859229 6084 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46667
I20260926 23:23:49.859321 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46667 every 8 connection(s)
I20260926 23:23:49.860585 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:49.869925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 6084
I20260926 23:23:49.870146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:49.870884 388 external_mini_cluster.cc:1428] Setting key d14a5f659e4920e2bb791600f54bb3e9
Sep 26 23:23:49 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:49.882177 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:49.863140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34537 (local address 127.0.97.62:46881)
0926 23:23:49.863608 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:49.863613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:49.864454 (+ 841us) server_negotiation.cc:407] Connection header received
0926 23:23:49.866304 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:49.866309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:49.866405 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:49.866523 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:49.867933 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:49.868612 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:49.870827 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:49.871301 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:49.871406 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:49.875304 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:49.875312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:49.875320 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:49.875361 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:49.878869 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:49.879819 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:49.879825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:49.879830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:49.879993 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:49.880392 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:49.880400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:49.880405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:49.880738 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:49.880863 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:49.881754 (+ 891us) server_negotiation.cc:299] Negotiation successful
0926 23:23:49.881933 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":285,"threads_started":1}
I20260926 23:23:49.883625 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:49.863599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46881 (local address 127.0.97.1:34537)
0926 23:23:49.864303 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:49.864344 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:49.866045 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:49.866746 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:49.866753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:49.867255 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:49.867770 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:49.867785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:49.868824 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:49.868831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:49.870614 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:49.870630 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:49.871516 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:49.871521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:49.871759 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:49.872492 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:49.872514 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:49.875078 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:49.879047 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:49.879058 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:49.879077 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:49.879684 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:49.880109 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:49.880112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:49.880114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:49.880283 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:49.880873 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:49.880884 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:49.881368 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 23:23:49.881676 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 23:23:49.882209 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":272,"threads_started":1}
I20260926 23:23:49.885120 6212 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46881
I20260926 23:23:49.885429 6212 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:49.886114 6212 heartbeater.cc:511] Master 127.0.97.62:46881 requested a full tablet report, sending...
I20260926 23:23:49.888118 6023 ts_manager.cc:194] Registered new tserver with Master: bea18957f9b64311ae5be614505443bd (127.0.97.1:46667)
I20260926 23:23:49.890337 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34537
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:23:49.920868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46881
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:50.235634 6218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:50.235911 6218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:50.236065 6218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:50.243014 6218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:50.243108 6218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:50.243326 6218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:50.252848 6218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:50.254196 6218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:50.255446 6218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:50.270426 6224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:50.270454 6226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:50.270519 6223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:50.271289 6218 server_base.cc:1034] running on GCE node
I20260926 23:23:50.271831 6218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:50.272742 6218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:50.273957 6218 hybrid_clock.cc:648] HybridClock initialized: now 1790465030273901 us; error 53 us; skew 500 ppm
I20260926 23:23:50.274261 6218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.277486 6218 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:23:50.278949 6218 webserver.cc:460] Webserver started at http://127.0.97.2:35845/ using document root <none> and password file <none>
I20260926 23:23:50.279596 6218 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:50.279721 6218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:50.280033 6218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:50.283715 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "01399aac1eb54aa38dca1f25ec636dd4"
format_stamp: "Formatted at 2026-09-26 23:23:50 on dist-test-slave-q5g8"
server_key: "84e10d86288b586a89c6b8aaed06e36e"
server_key_iv: "436a7b82de3eb268fe4b29cdb68ff93f"
server_key_version: "encryptionkey@0"
I20260926 23:23:50.284591 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "01399aac1eb54aa38dca1f25ec636dd4"
format_stamp: "Formatted at 2026-09-26 23:23:50 on dist-test-slave-q5g8"
server_key: "84e10d86288b586a89c6b8aaed06e36e"
server_key_iv: "436a7b82de3eb268fe4b29cdb68ff93f"
server_key_version: "encryptionkey@0"
I20260926 23:23:50.289602 6218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 23:23:50.293013 6233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:50.294478 6218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:23:50.294673 6218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "01399aac1eb54aa38dca1f25ec636dd4"
format_stamp: "Formatted at 2026-09-26 23:23:50 on dist-test-slave-q5g8"
server_key: "84e10d86288b586a89c6b8aaed06e36e"
server_key_iv: "436a7b82de3eb268fe4b29cdb68ff93f"
server_key_version: "encryptionkey@0"
I20260926 23:23:50.294906 6218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:50.313094 6218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:50.317044 6218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:50.317415 6218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:50.318455 6218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:50.321493 6218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:50.321589 6218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:50.321719 6218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:50.321779 6218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:50.356552 6218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46479
I20260926 23:23:50.356606 6345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46479 every 8 connection(s)
I20260926 23:23:50.357883 6218 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:50.359339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 6218
I20260926 23:23:50.359539 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:50.359961 388 external_mini_cluster.cc:1428] Setting key aecb27ac02a17240a3ec9280c72cc944
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:50.380336 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.360432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56277 (local address 127.0.97.62:46881)
0926 23:23:50.360761 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.360768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.362056 (+ 1288us) server_negotiation.cc:407] Connection header received
0926 23:23:50.363782 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.363790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.363878 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.363990 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.365459 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.366592 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.368579 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.369023 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.369124 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.373682 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:50.373687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:50.373691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:50.373726 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:50.376768 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.377887 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.377891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.377893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.378009 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.378539 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.378542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.378544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.378824 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:50.378931 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:50.379880 (+ 949us) server_negotiation.cc:299] Negotiation successful
0926 23:23:50.380062 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:23:50.381815 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.361046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46881 (local address 127.0.97.2:56277)
0926 23:23:50.361901 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:50.361950 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:50.363524 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:50.364184 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:50.364191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:50.364676 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:50.365296 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:50.365313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.366738 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.366742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:50.368410 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:50.368427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.369246 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.369255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:50.369578 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.370289 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:50.370323 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:50.373393 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:50.376956 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:50.376969 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:50.376992 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:50.377730 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:50.378127 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:50.378134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:50.378140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:50.378411 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:50.378958 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:50.378969 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:50.379460 (+ 491us) client_negotiation.cc:769] Sending connection context
0926 23:23:50.379796 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 23:23:50.380425 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":304,"threads_started":1}
I20260926 23:23:50.383163 6346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46881
I20260926 23:23:50.383548 6346 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:23:50.384315 6346 heartbeater.cc:511] Master 127.0.97.62:46881 requested a full tablet report, sending...
I20260926 23:23:50.385939 6023 ts_manager.cc:194] Registered new tserver with Master: 01399aac1eb54aa38dca1f25ec636dd4 (127.0.97.2:46479)
I20260926 23:23:50.387169 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:23:50.408195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46881
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:50.725787 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:50.726087 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:50.726301 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:50.733839 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:50.733965 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:50.734154 6351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:50.744256 6351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:50.745654 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:50.747094 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:50.760272 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:50.760387 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:50.760305 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:50.761059 6351 server_base.cc:1034] running on GCE node
I20260926 23:23:50.761819 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:50.762717 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:50.763969 6351 hybrid_clock.cc:648] HybridClock initialized: now 1790465030763917 us; error 42 us; skew 500 ppm
I20260926 23:23:50.764330 6351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.767956 6351 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:23:50.769547 6351 webserver.cc:460] Webserver started at http://127.0.97.3:40505/ using document root <none> and password file <none>
I20260926 23:23:50.770251 6351 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:50.770394 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:50.770709 6351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:50.774339 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "c47f7e005f7f4616829ec0db7c3bbc19"
format_stamp: "Formatted at 2026-09-26 23:23:50 on dist-test-slave-q5g8"
server_key: "584ce83c14887b56c534f8eec460d59c"
server_key_iv: "fe2425c3c9ee65dbd65d37c32781da82"
server_key_version: "encryptionkey@0"
I20260926 23:23:50.775152 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c47f7e005f7f4616829ec0db7c3bbc19"
format_stamp: "Formatted at 2026-09-26 23:23:50 on dist-test-slave-q5g8"
server_key: "584ce83c14887b56c534f8eec460d59c"
server_key_iv: "fe2425c3c9ee65dbd65d37c32781da82"
server_key_version: "encryptionkey@0"
I20260926 23:23:50.780649 6351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:23:50.784080 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:50.785491 6351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:50.785642 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "c47f7e005f7f4616829ec0db7c3bbc19"
format_stamp: "Formatted at 2026-09-26 23:23:50 on dist-test-slave-q5g8"
server_key: "584ce83c14887b56c534f8eec460d59c"
server_key_iv: "fe2425c3c9ee65dbd65d37c32781da82"
server_key_version: "encryptionkey@0"
I20260926 23:23:50.785813 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:50.811168 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:50.815925 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:50.816200 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:50.817173 6351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:50.818457 6351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:50.818528 6351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:50.818609 6351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:50.818662 6351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:50.851574 6351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33011
I20260926 23:23:50.851616 6478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33011 every 8 connection(s)
I20260926 23:23:50.852975 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:50.858101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 6351
I20260926 23:23:50.858309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:50.858880 388 external_mini_cluster.cc:1428] Setting key 7266c2163ea2517cef1ed2c4ee4affb6
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:23:50.873179 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.855535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53559 (local address 127.0.97.62:46881)
0926 23:23:50.855718 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.855722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.856885 (+ 1163us) server_negotiation.cc:407] Connection header received
0926 23:23:50.858902 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.858907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.859001 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.859127 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.861094 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.862237 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.863860 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.864250 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.864379 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.867937 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:50.867942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:50.867946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:50.867988 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:50.870332 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.870985 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.870989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.870991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.871109 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.871477 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.871480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.871482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.871736 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:50.871875 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:50.872568 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 23:23:50.872856 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:23:50.874145 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.855941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46881 (local address 127.0.97.3:53559)
0926 23:23:50.856728 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:50.856779 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:50.858555 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:50.859400 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:50.859411 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:50.860151 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:50.860922 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:50.860948 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.862386 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.862392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:50.863704 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:50.863715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.864481 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.864488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:50.864804 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.865455 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:50.865481 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:50.867723 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:50.870456 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:50.870463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:50.870476 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:50.870870 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:50.871206 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:50.871209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:50.871211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:50.871381 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:50.871868 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:50.871874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:50.872170 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 23:23:50.872448 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 23:23:50.872988 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":280,"threads_started":1}
I20260926 23:23:50.875504 6479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46881
I20260926 23:23:50.875808 6479 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:50.876520 6479 heartbeater.cc:511] Master 127.0.97.62:46881 requested a full tablet report, sending...
I20260926 23:23:50.877945 6023 ts_manager.cc:194] Registered new tserver with Master: c47f7e005f7f4616829ec0db7c3bbc19 (127.0.97.3:33011)
I20260926 23:23:50.878794 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53559
I20260926 23:23:50.886739 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:23:50.894042 6212 heartbeater.cc:503] Master 127.0.97.62:46881 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/test-admin.keytab.
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.914903 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 23:23:50.920058 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:50.917645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.0.97.1:46667)
0926 23:23:50.918246 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.918253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.918303 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:23:50.918363 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.918370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.918581 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.918707 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.919575 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55660: BlockingRecv error: recv got EOF from 127.0.0.1:55660 (error 108)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":298,"threads_started":1}
W20260926 23:23:50.920096 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.921784 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.923147 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:50.939977 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.924965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.0.97.1:46667)
0926 23:23:50.925204 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.925212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.925733 (+ 521us) server_negotiation.cc:407] Connection header received
0926 23:23:50.925955 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.925959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.926026 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.926130 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.927128 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.928093 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.929382 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.929934 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.930075 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.934030 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:50.934043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:50.934059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:50.934105 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:50.937159 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.937885 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.937893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.937896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.938017 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.938437 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.938448 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.938451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.938803 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:50.939091 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:50.939431 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 23:23:50.939682 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 23:23:50.946030 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:50.944212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.0.97.1:46667)
0926 23:23:50.944625 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.944630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.944646 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:23:50.944736 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.944741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.944929 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.945008 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.945751 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55680: BlockingRecv error: recv got EOF from 127.0.0.1:55680 (error 108)
Metrics: {"server-negotiator.queue_time_us":298}
W20260926 23:23:50.946122 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.947607 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.948945 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:50.962905 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.950121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.0.97.1:46667)
0926 23:23:50.950345 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.950350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.950551 (+ 201us) server_negotiation.cc:407] Connection header received
0926 23:23:50.950682 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.950685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.950739 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.950809 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.951667 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.952341 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.953558 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.953974 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.954112 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.957765 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:50.957770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:50.957773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:50.957805 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:50.960546 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.961253 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.961258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.961260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.961374 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.961701 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.961704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.961706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.961947 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:50.962133 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:50.962438 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 23:23:50.962627 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 23:23:50.967185 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:23:50.965644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.97.1:46667)
0926 23:23:50.965838 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.965845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.965986 (+ 141us) server_negotiation.cc:407] Connection header received
0926 23:23:50.966110 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.966114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.966181 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.966264 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.966963 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55702: BlockingRecv error: recv got EOF from 127.0.0.1:55702 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 23:23:50.967355 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790464978839002-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.968947 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:50.970237 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:23:50 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:23:50.985428 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.971647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.97.1:46667)
0926 23:23:50.971842 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.971850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.972022 (+ 172us) server_negotiation.cc:407] Connection header received
0926 23:23:50.972177 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.972181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.972243 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.972332 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.973560 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.974325 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.975649 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.976159 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.976300 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.980121 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:50.980126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:50.980130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:50.980170 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:50.982756 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.983478 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.983490 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.983493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.983616 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.984007 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.984014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.984017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.984300 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:50.984633 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:50.984895 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 23:23:50.985102 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:23:50.987110 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 6084
I20260926 23:23:50.999163 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:50.987792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.0.97.1:46667)
0926 23:23:50.987982 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:50.987990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:50.988320 (+ 330us) server_negotiation.cc:407] Connection header received
0926 23:23:50.988521 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:50.988525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:50.988579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:50.988676 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:50.989699 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.990427 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.991600 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:50.992009 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:50.992140 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:50.994104 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:50.994109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:50.994112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:50.994141 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:50.996417 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.997144 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.997150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.997153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.997291 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:50.997681 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:50.997686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:50.997688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:50.997957 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:50.998255 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:50.998582 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 23:23:50.998804 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 23:23:50.999873 6207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:51.082052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 6084
I20260926 23:23:51.098703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 6218
Sep 26 23:23:51 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:23:51.115339 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:51.099665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.0.97.2:46479)
0926 23:23:51.100232 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:51.100245 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:51.100270 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:23:51.100338 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:51.100344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:51.100548 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:51.100744 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:51.102013 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:51.103077 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:51.104385 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:51.105092 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:51.105252 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:51.109299 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:51.109313 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:51.109328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:51.109372 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:51.112464 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:51.113438 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:51.113445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:51.113448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:51.113587 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:51.113955 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:51.113958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:51.113960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:51.114312 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:51.114521 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:51.114765 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 23:23:51.115016 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":313,"threads_started":1}
I20260926 23:23:51.116598 6341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:51.198588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 6218
I20260926 23:23:51.214897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 6351
Sep 26 23:23:51 dist-test-slave-q5g8 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:23:51.231137 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:51.215875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.0.97.3:33011)
0926 23:23:51.216482 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:51.216489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:51.216517 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:23:51.216577 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:51.216583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:51.216881 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:51.217012 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:51.218008 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:51.219013 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:51.220254 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:51.220907 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:51.221005 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:51.225029 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:51.225043 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:51.225060 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:51.225102 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:51.228189 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:51.229079 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:51.229088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:51.229090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:51.229226 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:51.229588 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:51.229591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:51.229593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:51.229943 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:51.230198 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:51.230526 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 23:23:51.230820 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":341,"threads_started":1}
I20260926 23:23:51.232543 6474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:51.323791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 6351
I20260926 23:23:51.339371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 5991
I20260926 23:23:51.340730 6053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:51.442842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 5991
2026-09-26T23:23:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3936 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:23:51 dist-test-slave-q5g8 krb5kdc[6506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:23:51 dist-test-slave-q5g8 krb5kdc[6506](info): set up 2 sockets
Sep 26 23:23:51 dist-test-slave-q5g8 krb5kdc[6506](info): commencing operation
krb5kdc: starting...
W20260926 23:23:52.617623 388 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:23:52 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:23:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 23:23:52.849364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33391
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:53.199031 6519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:53.199378 6519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:53.199550 6519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:53.207876 6519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:53.208010 6519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:53.208101 6519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:53.208191 6519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:53.208253 6519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:53.218698 6519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:53.220263 6519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:53.221791 6519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:53.235184 6524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:53.235296 6525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:53.235250 6527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:53.236668 6519 server_base.cc:1034] running on GCE node
I20260926 23:23:53.237576 6519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:53.239137 6519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:53.240412 6519 hybrid_clock.cc:648] HybridClock initialized: now 1790465033240347 us; error 42 us; skew 500 ppm
I20260926 23:23:53.240875 6519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:53 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:53.244570 6519 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 23:23:53.246361 6519 webserver.cc:460] Webserver started at http://127.0.97.62:43863/ using document root <none> and password file <none>
I20260926 23:23:53.247102 6519 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:53.247213 6519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:53.247488 6519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:53.250744 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "62fec46c64854df8b05a61a9b3831283"
format_stamp: "Formatted at 2026-09-26 23:23:53 on dist-test-slave-q5g8"
server_key: "32830717badea5021dff5602e542609e"
server_key_iv: "ff97a337207ea748cd4249ed4deee312"
server_key_version: "encryptionkey@0"
I20260926 23:23:53.251612 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "62fec46c64854df8b05a61a9b3831283"
format_stamp: "Formatted at 2026-09-26 23:23:53 on dist-test-slave-q5g8"
server_key: "32830717badea5021dff5602e542609e"
server_key_iv: "ff97a337207ea748cd4249ed4deee312"
server_key_version: "encryptionkey@0"
I20260926 23:23:53.257332 6519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 23:23:53.261073 6534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:53.262591 6519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:53.262748 6519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "62fec46c64854df8b05a61a9b3831283"
format_stamp: "Formatted at 2026-09-26 23:23:53 on dist-test-slave-q5g8"
server_key: "32830717badea5021dff5602e542609e"
server_key_iv: "ff97a337207ea748cd4249ed4deee312"
server_key_version: "encryptionkey@0"
I20260926 23:23:53.262935 6519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:53.284117 6519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:53.288873 6519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:53.289134 6519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:53.310395 6519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33391
I20260926 23:23:53.310441 6585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33391 every 8 connection(s)
I20260926 23:23:53.311933 6519 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:23:53.312371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 6519
I20260926 23:23:53.312623 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:23:53.313563 388 external_mini_cluster.cc:1428] Setting key 18a92d3d90f48f2837d57c28cf684ab4
I20260926 23:23:53.317577 6586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:23:53 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:23:53.332708 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: Bootstrap starting.
I20260926 23:23:53.336431 6586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:53.337841 6586 log.cc:824] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:53.339161 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:53.315783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.0.97.62:33391)
0926 23:23:53.316597 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:53.316610 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:53.316652 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:23:53.318486 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:53.318510 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:53.318905 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:53.319247 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:53.321295 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:53.322678 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:53.324764 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:53.325476 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:53.325617 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:53.330877 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:53.330886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:53.330905 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:53.330955 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:53.334282 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:53.335132 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:53.335141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:53.335148 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:53.335327 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:53.335765 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:53.335770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:53.335773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:53.336419 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:53.336769 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:53.337286 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 23:23:53.337741 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":377,"threads_started":1}
I20260926 23:23:53.341439 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: No bootstrap required, opened a new log
I20260926 23:23:53.350652 6586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260926 23:23:53.351116 6586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:53.351243 6586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fec46c64854df8b05a61a9b3831283, State: Initialized, Role: FOLLOWER
I20260926 23:23:53.351935 6586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260926 23:23:53.352205 6586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:23:53.352373 6586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:23:53.352581 6586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:53.355180 6586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260926 23:23:53.355764 6586 leader_election.cc:302] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62fec46c64854df8b05a61a9b3831283; no voters:
I20260926 23:23:53.356418 6586 leader_election.cc:288] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:23:53.356560 6591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:53.357514 6591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 1 LEADER]: Becoming Leader. State: Replica: 62fec46c64854df8b05a61a9b3831283, State: Running, Role: LEADER
I20260926 23:23:53.358057 6591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260926 23:23:53.358963 6586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:53.361018 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } }
I20260926 23:23:53.361441 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:53.361037 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fec46c64854df8b05a61a9b3831283. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } }
I20260926 23:23:53.361593 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:53.364919 6597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:23:53.370977 6597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:23:53.382004 6597 catalog_manager.cc:1347] Generated new cluster ID: 35e0431d34e74cdfa6af1b055132117a
I20260926 23:23:53.382148 6597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:23:53.397984 6597 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:23:53.399501 6597 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:23:53.408270 6597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: Generated new TSK 0
I20260926 23:23:53.409371 6597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 23:23:53.481236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33391
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:53.831743 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:53.832037 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:53.832206 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:53.839654 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:53.839785 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:53.840014 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:53.850788 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:53.852377 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:53.853921 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:53.868291 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:53.868299 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:53.868393 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:53.868938 6612 server_base.cc:1034] running on GCE node
I20260926 23:23:53.869728 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:53.870784 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:53.872002 6612 hybrid_clock.cc:648] HybridClock initialized: now 1790465033871925 us; error 68 us; skew 500 ppm
I20260926 23:23:53.872341 6612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:53 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:53.875942 6612 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:23:53.877840 6612 webserver.cc:460] Webserver started at http://127.0.97.1:32775/ using document root <none> and password file <none>
I20260926 23:23:53.878713 6612 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:53.878901 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:53.879259 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:53.883009 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "070ebf88cc1b44e49719cc523a51e250"
format_stamp: "Formatted at 2026-09-26 23:23:53 on dist-test-slave-q5g8"
server_key: "ea3c7ce6fc4fc493e680f9231d2671f3"
server_key_iv: "63380ed86a193ca4513b4f0b93860cb7"
server_key_version: "encryptionkey@0"
I20260926 23:23:53.883991 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "070ebf88cc1b44e49719cc523a51e250"
format_stamp: "Formatted at 2026-09-26 23:23:53 on dist-test-slave-q5g8"
server_key: "ea3c7ce6fc4fc493e680f9231d2671f3"
server_key_iv: "63380ed86a193ca4513b4f0b93860cb7"
server_key_version: "encryptionkey@0"
I20260926 23:23:53.889871 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 23:23:53.893961 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:53.895502 6612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260926 23:23:53.895717 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "070ebf88cc1b44e49719cc523a51e250"
format_stamp: "Formatted at 2026-09-26 23:23:53 on dist-test-slave-q5g8"
server_key: "ea3c7ce6fc4fc493e680f9231d2671f3"
server_key_iv: "63380ed86a193ca4513b4f0b93860cb7"
server_key_version: "encryptionkey@0"
I20260926 23:23:53.895987 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:53.909969 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:53.914415 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:53.914734 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:53.915750 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:53.917209 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:53.917316 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:53.917475 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:53.917570 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:53.954953 6612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39011
I20260926 23:23:53.954994 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39011 every 8 connection(s)
I20260926 23:23:53.956409 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:53.963915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 6612
I20260926 23:23:53.964157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:23:53.965018 388 external_mini_cluster.cc:1428] Setting key c01656ccd665eeb9ccaad309370c5bd9
Sep 26 23:23:53 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:23:53.983140 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:53.959255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58705 (local address 127.0.97.62:33391)
0926 23:23:53.959719 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:53.959724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:53.960956 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 23:23:53.962849 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:53.962855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:53.962953 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:53.963074 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:53.965328 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:53.966405 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:53.970154 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:53.970608 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:53.970739 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:53.975800 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:53.975810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:53.975817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:53.975865 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:53.979543 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:53.980399 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:53.980408 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:53.980415 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:53.980567 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:53.981075 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:53.981083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:53.981089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:53.981470 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:53.981646 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:53.982364 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 23:23:53.982648 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260926 23:23:53.983807 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:53.959941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.1:58705)
0926 23:23:53.960797 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:53.960839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:53.962557 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:53.963329 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:53.963338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:53.964162 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:53.965102 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:53.965132 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:53.966624 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:53.966630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:53.968878 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:53.968904 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:53.970902 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:53.970912 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:53.971279 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:53.972141 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:53.972177 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:53.975460 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:53.979758 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:53.979767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:53.979781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:53.980245 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:53.980676 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:53.980748 (+ 72us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:53.980753 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:53.980946 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:53.981592 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:53.981598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:53.981949 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 23:23:53.982200 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 23:23:53.982630 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":285,"threads_started":1}
I20260926 23:23:53.985587 6740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33391
I20260926 23:23:53.985922 6740 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:53.986749 6740 heartbeater.cc:511] Master 127.0.97.62:33391 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:23:53.989082 6551 ts_manager.cc:194] Registered new tserver with Master: 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011)
I20260926 23:23:53.990655 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 23:23:54.018705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33391
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:54.378508 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:54.378813 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:54.378970 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:54.386404 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:54.386528 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:54.386643 6746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:54.396744 6746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:54.398073 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:54.399583 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:54.413755 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:54.413817 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:54.413755 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:54.414785 6746 server_base.cc:1034] running on GCE node
I20260926 23:23:54.415421 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:54.416440 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:54.417641 6746 hybrid_clock.cc:648] HybridClock initialized: now 1790465034417586 us; error 54 us; skew 500 ppm
I20260926 23:23:54.417932 6746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:54 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:54.430331 6746 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:23:54.432242 6746 webserver.cc:460] Webserver started at http://127.0.97.2:43549/ using document root <none> and password file <none>
I20260926 23:23:54.433111 6746 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:54.433238 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:54.433521 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:54.436846 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "68a492260d0b4fb881a25623fde1b624"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "27693b6600a001fe9631faaf90112eb0"
server_key_iv: "aabfae9336a36c6c17a3f6b535ce212f"
server_key_version: "encryptionkey@0"
I20260926 23:23:54.437731 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "68a492260d0b4fb881a25623fde1b624"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "27693b6600a001fe9631faaf90112eb0"
server_key_iv: "aabfae9336a36c6c17a3f6b535ce212f"
server_key_version: "encryptionkey@0"
I20260926 23:23:54.443570 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:23:54.447429 6761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:54.449230 6746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:54.449422 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "68a492260d0b4fb881a25623fde1b624"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "27693b6600a001fe9631faaf90112eb0"
server_key_iv: "aabfae9336a36c6c17a3f6b535ce212f"
server_key_version: "encryptionkey@0"
I20260926 23:23:54.449630 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:54.475723 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:54.480813 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:54.481158 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:54.482316 6746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:54.483903 6746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:54.483985 6746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:54.484148 6746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:54.484207 6746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:54.522198 6746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37491
I20260926 23:23:54.522322 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37491 every 8 connection(s)
I20260926 23:23:54.523594 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:54.532176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 6746
I20260926 23:23:54.532428 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:23:54.533057 388 external_mini_cluster.cc:1428] Setting key 0d43114c2a8a2bd4bc1bd085ba3b049a
Sep 26 23:23:54 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:23:54.546113 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:54.526534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34489 (local address 127.0.97.62:33391)
0926 23:23:54.526986 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:54.526991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:54.528073 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 23:23:54.530065 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:54.530072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:54.530173 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:54.530297 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:54.531938 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:54.532856 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:54.534692 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:54.535281 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:54.535435 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:54.539529 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:54.539535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:54.539538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:54.539573 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:54.542426 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:54.543275 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:54.543282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:54.543286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:54.543502 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:54.544148 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:54.544152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:54.544154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:54.544443 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:54.544562 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:54.545651 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0926 23:23:54.545841 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":241,"threads_started":1}
I20260926 23:23:54.547864 6877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:54.527103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.2:34489)
0926 23:23:54.527874 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:54.527940 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:54.529749 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:54.530570 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:54.530580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:54.531128 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:54.531724 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:54.531742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:54.533023 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:54.533027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:54.534529 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:54.534543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:54.535542 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:54.535549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:54.535821 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:54.536580 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:54.536604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:54.539216 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:54.542607 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:54.542617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:54.542644 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:54.543132 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:54.543670 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:54.543679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:54.543686 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:54.544007 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:54.544628 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:54.544640 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:54.545196 (+ 556us) client_negotiation.cc:769] Sending connection context
0926 23:23:54.545542 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 23:23:54.546206 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":315,"threads_started":1}
I20260926 23:23:54.549526 6874 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33391
I20260926 23:23:54.549896 6874 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:54.550664 6874 heartbeater.cc:511] Master 127.0.97.62:33391 requested a full tablet report, sending...
I20260926 23:23:54.552353 6551 ts_manager.cc:194] Registered new tserver with Master: 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491)
I20260926 23:23:54.553352 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34489
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 23:23:54.585516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33391
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:54.932710 6880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:54.933105 6880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:54.933293 6880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:54.941406 6880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:54.941615 6880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:54.941843 6880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:54.952581 6880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:54.954366 6880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:54.956352 6880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:54.972546 6885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:54.972643 6888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:54.972712 6886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:54.973701 6880 server_base.cc:1034] running on GCE node
I20260926 23:23:54.974470 6880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:54.975600 6880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:54.976824 6880 hybrid_clock.cc:648] HybridClock initialized: now 1790465034976766 us; error 54 us; skew 500 ppm
I20260926 23:23:54.977301 6880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:54 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:54.981026 6880 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:23:54.982789 6880 webserver.cc:460] Webserver started at http://127.0.97.3:43161/ using document root <none> and password file <none>
I20260926 23:23:54.983634 6880 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:54.983801 6880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:54.984179 6880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:54.987564 6880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "2fccafc1129343ae9e66789b30ee0f66"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "15385fef2004bdfdb0a3457a4f9be8ac"
server_key_iv: "b510c2e4c980b745b3b6793ae7cf97c3"
server_key_version: "encryptionkey@0"
I20260926 23:23:54.988402 6880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2fccafc1129343ae9e66789b30ee0f66"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "15385fef2004bdfdb0a3457a4f9be8ac"
server_key_iv: "b510c2e4c980b745b3b6793ae7cf97c3"
server_key_version: "encryptionkey@0"
I20260926 23:23:54.993927 6740 heartbeater.cc:503] Master 127.0.97.62:33391 was elected leader, sending a full tablet report...
I20260926 23:23:54.993922 6880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 23:23:54.997814 6895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:54.999248 6880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:23:54.999392 6880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "2fccafc1129343ae9e66789b30ee0f66"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "15385fef2004bdfdb0a3457a4f9be8ac"
server_key_iv: "b510c2e4c980b745b3b6793ae7cf97c3"
server_key_version: "encryptionkey@0"
I20260926 23:23:54.999579 6880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:55.013484 6880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:55.017907 6880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:55.018153 6880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:55.019114 6880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:55.020445 6880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:55.020553 6880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:55.020642 6880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:55.020736 6880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:55.057399 6880 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41405
I20260926 23:23:55.057502 7007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41405 every 8 connection(s)
I20260926 23:23:55.058879 6880 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:55.059114 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 6880
I20260926 23:23:55.059330 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:23:55.060038 388 external_mini_cluster.cc:1428] Setting key 3f1275c50a2e97d79a896f5065b1c286
Sep 26 23:23:55 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465034, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:23:55.081384 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.062321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32847 (local address 127.0.97.62:33391)
0926 23:23:55.062977 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.062984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.064110 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 23:23:55.066329 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.066333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.066438 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.066607 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.068336 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.069260 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.070886 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.071319 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.071426 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.075237 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.075243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.075246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.075280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.078399 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.079164 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.079168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.079171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.079316 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.079717 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.079721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.079723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.080005 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.080120 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.080896 (+ 776us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.081086 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":1056,"threads_started":1}
I20260926 23:23:55.082486 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.062989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:32847)
0926 23:23:55.063930 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:55.063982 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:55.066017 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:55.066862 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:55.066872 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:55.067443 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:55.068134 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.068152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.069424 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.069428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.070730 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.070748 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.071522 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.071527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.071727 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.072398 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:55.072422 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:55.074999 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:55.078546 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.078554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.078571 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.079048 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.079417 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.079420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.079422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.079619 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.080117 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:55.080123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:55.080475 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 23:23:55.080812 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 23:23:55.081236 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":398,"threads_started":1}
I20260926 23:23:55.084129 7008 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33391
I20260926 23:23:55.084518 7008 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:55.085425 7008 heartbeater.cc:511] Master 127.0.97.62:33391 requested a full tablet report, sending...
I20260926 23:23:55.087008 6551 ts_manager.cc:194] Registered new tserver with Master: 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405)
I20260926 23:23:55.087859 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32847
I20260926 23:23:55.089115 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260926 23:23:55.141537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40969
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33391,127.0.97.61:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:55.482774 7014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:55.483153 7014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:55.483309 7014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:55.490993 7014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:55.491159 7014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:55.491277 7014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:55.491382 7014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:55.491466 7014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:55.502832 7014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33391,127.0.97.61:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
W20260926 23:23:55.503334 7014 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:33391,127.0.97.61:40969'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 23:23:55.504750 7014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:55.506251 7014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:55.518882 7020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:55.518839 7022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:55.518903 7019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:55.519630 7014 server_base.cc:1034] running on GCE node
I20260926 23:23:55.520463 7014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:55.522215 7014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:55.523461 7014 hybrid_clock.cc:648] HybridClock initialized: now 1790465035523389 us; error 61 us; skew 500 ppm
I20260926 23:23:55.523825 7014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:55 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:55.527100 7014 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 23:23:55.528792 7014 webserver.cc:460] Webserver started at http://127.0.97.61:36181/ using document root <none> and password file <none>
I20260926 23:23:55.529493 7014 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:55.529587 7014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:55.529871 7014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:55.533110 7014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/instance:
uuid: "1b16402ace3444f2ba0b2a37688bcf9d"
format_stamp: "Formatted at 2026-09-26 23:23:55 on dist-test-slave-q5g8"
server_key: "1f58d6b13ad832b63b5ac65eaa2d1e55"
server_key_iv: "a7dcedcdb2357631d40df6fe800be281"
server_key_version: "encryptionkey@0"
I20260926 23:23:55.533919 7014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal/instance:
uuid: "1b16402ace3444f2ba0b2a37688bcf9d"
format_stamp: "Formatted at 2026-09-26 23:23:55 on dist-test-slave-q5g8"
server_key: "1f58d6b13ad832b63b5ac65eaa2d1e55"
server_key_iv: "a7dcedcdb2357631d40df6fe800be281"
server_key_version: "encryptionkey@0"
I20260926 23:23:55.539620 7014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:23:55.543937 7029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:55.545739 7014 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:23:55.545975 7014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
uuid: "1b16402ace3444f2ba0b2a37688bcf9d"
format_stamp: "Formatted at 2026-09-26 23:23:55 on dist-test-slave-q5g8"
server_key: "1f58d6b13ad832b63b5ac65eaa2d1e55"
server_key_iv: "a7dcedcdb2357631d40df6fe800be281"
server_key_version: "encryptionkey@0"
I20260926 23:23:55.546239 7014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:55.556164 6874 heartbeater.cc:503] Master 127.0.97.62:33391 was elected leader, sending a full tablet report...
I20260926 23:23:55.577276 7014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:55.581969 7014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:55.582297 7014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:55.602028 7014 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40969
I20260926 23:23:55.602072 7080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40969 every 8 connection(s)
I20260926 23:23:55.603575 7014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:23:55.607674 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:55.613214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 7014
I20260926 23:23:55.614506 7081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:23:55 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 23:23:55 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:23:55.635936 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.617545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.0.97.62:33391)
0926 23:23:55.618052 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.618058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.618653 (+ 595us) server_negotiation.cc:407] Connection header received
0926 23:23:55.619349 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.619353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.619441 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.619555 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.621113 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.621909 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.623723 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.624218 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.624336 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.628423 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.628428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.628432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.628506 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.632065 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.632992 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.633001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.633007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.633161 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.633772 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.633780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.633785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.634248 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.634415 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.635330 (+ 915us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.635575 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260926 23:23:55.637071 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.615646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.0.97.61:40969)
0926 23:23:55.616462 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.616473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.616516 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:23:55.619071 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.619100 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.619431 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.619654 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.621136 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.622538 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.623893 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.624577 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.624839 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.629515 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.629527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.629545 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.629599 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.633365 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.634153 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.634161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.634164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.634305 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.634706 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.634709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.634711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.635064 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.635347 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.635685 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.635925 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"spinlock_wait_cycles":52352,"thread_start_us":354,"threads_started":1}
I20260926 23:23:55.637112 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.617591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:42482)
0926 23:23:55.618472 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:55.618523 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:55.619071 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:55.619654 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:55.619661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:55.620450 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:55.620904 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.620929 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.622058 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.622064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.623544 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.623556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.624473 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.624480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.624734 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.625656 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:55.625694 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:55.628198 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:55.632247 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.632260 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.632270 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.632859 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.633322 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.633326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.633328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.633605 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.634420 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:55.634428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:55.634905 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 23:23:55.635201 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 23:23:55.635790 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":409,"threads_started":1}
I20260926 23:23:55.638953 7081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:23:55 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:23:55.650960 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.639653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.0.1:51052)
0926 23:23:55.639865 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:55.639885 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:55.640008 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:55.640320 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:55.640323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:55.640625 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:55.640819 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.640827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.641711 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.641714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.642577 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.642587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.643287 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.643290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.643440 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.644053 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:55.644070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:55.646143 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:55.648914 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.648917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.648920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.649253 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.649608 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.649611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.649613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.649723 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.650178 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:55.650181 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:55.650316 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 23:23:55.650462 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 23:23:55.650686 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 23:23:55.651163 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.639827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.0.97.61:40969)
0926 23:23:55.639991 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.639996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.640014 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:23:55.640122 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.640126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.640199 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.640276 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.640933 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.641608 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.642766 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.643160 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.643264 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.646334 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.646340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.646343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.646374 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.648799 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.649347 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.649351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.649353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.649499 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.649809 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.649812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.649814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.650084 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.650219 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.650531 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.650798 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":45}
I20260926 23:23:55.655951 7081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Bootstrap starting.
I20260926 23:23:55.659281 7081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:55.660408 7081 log.cc:824] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:55.663072 7081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: No bootstrap required, opened a new log
I20260926 23:23:55.670619 7081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } }
I20260926 23:23:55.670955 7081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:55.671077 7081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b16402ace3444f2ba0b2a37688bcf9d, State: Initialized, Role: FOLLOWER
I20260926 23:23:55.671793 7081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } }
I20260926 23:23:55.672621 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } } }
I20260926 23:23:55.672998 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:23:55.673767 7081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:23:55.684041 7099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:23:55.684247 7099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:23:55.686213 7014 master_runner.cc:428] Detected that this master 1b16402ace3444f2ba0b2a37688bcf9d is joining an existing cluster
I20260926 23:23:55.686378 7014 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:40969
I20260926 23:23:55.712244 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.691394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.0.97.62:33391)
0926 23:23:55.692170 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.692181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.693189 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 23:23:55.693562 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.693572 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.693714 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.693804 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.695935 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.697495 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.700398 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.701286 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.701404 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.704368 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.704377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.704385 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.704433 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.708657 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.709579 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.709588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.709595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.709753 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.710238 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.710246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.710253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.710661 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.710791 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.711502 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.711777 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575}
I20260926 23:23:55.715019 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.691046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:42490)
0926 23:23:55.693040 (+ 1994us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:55.693069 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:55.693359 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:55.694090 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:55.694098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:55.695106 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:55.695667 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.695686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.697750 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.697759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.700128 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.700153 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.701706 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.701714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.701923 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.702672 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:55.702702 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:55.704092 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:55.708849 (+ 4757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.708857 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.708864 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.709422 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.709910 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.709918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.709928 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.710115 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.710892 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:55.710903 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:55.711169 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 23:23:55.712225 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0926 23:23:55.714410 (+ 2185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":21900,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 23:23:55.723994 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.692020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.0.1:51060)
0926 23:23:55.704497 (+ 12477us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:55.704523 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:55.704763 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:55.705358 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:55.705363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:55.706109 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:55.706467 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.706480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.708304 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.708312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.710162 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.710180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.711619 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.711627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.711837 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.712610 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:55.712638 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:55.713987 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:55.720396 (+ 6409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.720404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.720411 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.721007 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.721567 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.721572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.721575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.721745 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.722574 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:55.722579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:55.722864 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 23:23:55.723040 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 23:23:55.723419 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12187,"thread_start_us":4927,"threads_started":1}
I20260926 23:23:55.725814 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.696881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.97.61:40969)
0926 23:23:55.697073 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.697086 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.704611 (+ 7525us) server_negotiation.cc:407] Connection header received
0926 23:23:55.704951 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.704957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.705029 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.705119 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.706645 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.708053 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.710351 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.711166 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.711287 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.714224 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.714234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.714242 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.714287 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.719064 (+ 4777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.721174 (+ 2110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.721180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.721184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.721394 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.721898 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.721909 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.721913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.722331 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.722513 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.724888 (+ 2375us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.725233 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 26 23:23:55 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:23:55.753549 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.723392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.0.97.61:40969)
0926 23:23:55.727885 (+ 4493us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.727894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.732498 (+ 4604us) server_negotiation.cc:407] Connection header received
0926 23:23:55.732981 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.732985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.733081 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.733175 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.735070 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.736306 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.738472 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.739286 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.739468 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.744305 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.744314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.744319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.744367 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.748984 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.750237 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.750245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.750249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.750480 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.751195 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.751201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.751205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.751664 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.751843 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.752626 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.752939 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4318,"thread_start_us":8015,"threads_started":1}
I20260926 23:23:55.755170 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.722767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.0.1:51072)
0926 23:23:55.732296 (+ 9529us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:55.732359 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:55.732731 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:55.733284 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:55.733294 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:55.734481 (+ 1187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:55.734830 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.734849 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.736489 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.736496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.738274 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.738294 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.739516 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.739524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.739783 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.740734 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:55.740765 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:55.744022 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:55.749221 (+ 5199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.749244 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.749256 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.750044 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.750753 (+ 709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.750759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.750764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.751025 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.751959 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:55.751968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:55.752229 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 23:23:55.754146 (+ 1917us) client_negotiation.cc:240] Negotiation successful
0926 23:23:55.754678 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9195,"thread_start_us":9021,"threads_started":1}
I20260926 23:23:55.758368 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:40969: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 40969
}
attrs {
promote: true
}
}
dest_uuid: "62fec46c64854df8b05a61a9b3831283"
cas_config_opid_index: -1
I20260926 23:23:55.759658 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } }
I20260926 23:23:55.764098 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1b16402ace3444f2ba0b2a37688bcf9d (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:55.765436 7056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260926 23:23:55.769760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35185
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:23:55.770524 7056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Refusing update from remote peer 62fec46c64854df8b05a61a9b3831283: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 23:23:55.771924 7112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:55.786388 7113 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:40969
I20260926 23:23:55.787344 7014 master.cc:537] Master@127.0.97.61:40969 shutting down...
I20260926 23:23:55.791597 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:55.791992 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:55.836423 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1b16402ace3444f2ba0b2a37688bcf9d (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:55.842110 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:55.842567 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: LEARNER
I20260926 23:23:55.842886 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:55.843096 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: LEARNER
I20260926 23:23:55.842988 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fec46c64854df8b05a61a9b3831283. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:55.843250 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: LEARNER
I20260926 23:23:55.845950 7014 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 LEARNER]: Raft consensus shutting down.
I20260926 23:23:55.846498 7014 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 LEARNER]: Raft consensus is shut down!
I20260926 23:23:55.846671 7014 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: stopping tablet replica
I20260926 23:23:55.855880 7014 master.cc:552] Master@127.0.97.61:40969 shutdown complete.
I20260926 23:23:55.855988 7014 master_runner.cc:305] Clearing existing system tablet
I20260926 23:23:55.856451 7014 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:55.856554 7014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:55.859858 7014 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:55.861876 7124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:55.862287 7014 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:55.862752 7014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 23:23:55.862905 7014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
uuid: "1b16402ace3444f2ba0b2a37688bcf9d"
format_stamp: "Formatted at 2026-09-26 23:23:55 on dist-test-slave-q5g8"
server_key: "1f58d6b13ad832b63b5ac65eaa2d1e55"
server_key_iv: "a7dcedcdb2357631d40df6fe800be281"
server_key_version: "encryptionkey@0"
I20260926 23:23:55.868458 7014 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:55.876061 7014 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 23:23:55.876437 7014 log.cc:1196] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 23:23:55.877143 7014 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Deleting consensus metadata
I20260926 23:23:55.877493 7014 master_runner.cc:315] Copying system tablet from 127.0.97.62:33391
I20260926 23:23:55.882329 7014 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33391
I20260926 23:23:55.900043 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.883949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.0.97.62:33391)
0926 23:23:55.884161 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:55.884168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:55.884669 (+ 501us) server_negotiation.cc:407] Connection header received
0926 23:23:55.884972 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:55.884981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:55.885114 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:55.885280 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:55.886363 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.887558 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.889537 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.890198 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.890334 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.892719 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:55.892727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:55.892732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:55.892774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:55.896601 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.897557 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.897563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.897566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.897743 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:55.898191 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:55.898195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:55.898198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:55.898553 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:55.898739 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:55.899263 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 23:23:55.899586 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:23:55.900414 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:55.883364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:42504)
0926 23:23:55.884514 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:55.884536 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:55.884737 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:55.885317 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:55.885322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:55.885828 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:55.886151 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.886165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.887720 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.887726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.889366 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:55.889384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:55.890377 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:55.890384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:55.890579 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:55.891357 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:55.891380 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:55.892453 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:55.896823 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.896830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.896836 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.897409 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.897871 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:55.897876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:55.897880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:55.898062 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:55.898702 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:55.898707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:55.898902 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 23:23:55.899083 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 23:23:55.899436 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":825,"threads_started":1}
I20260926 23:23:55.901502 6571 tablet_copy_service.cc:140] P 62fec46c64854df8b05a61a9b3831283: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1b16402ace3444f2ba0b2a37688bcf9d ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42504)
I20260926 23:23:55.901827 6571 tablet_copy_service.cc:161] P 62fec46c64854df8b05a61a9b3831283: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1b16402ace3444f2ba0b2a37688bcf9d at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42504: session id = 1b16402ace3444f2ba0b2a37688bcf9d-00000000000000000000000000000000
I20260926 23:23:55.906930 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 23:23:55.909695 7014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 23:23:55.917400 7014 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: tablet copy: Starting download of 0 data blocks...
I20260926 23:23:55.917770 7014 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: tablet copy: Starting download of 1 WAL segments...
I20260926 23:23:55.920948 7014 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 23:23:55.925773 6571 tablet_copy_service.cc:342] P 62fec46c64854df8b05a61a9b3831283: Request end of tablet copy session 1b16402ace3444f2ba0b2a37688bcf9d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42504
I20260926 23:23:55.926033 6571 tablet_copy_service.cc:433] P 62fec46c64854df8b05a61a9b3831283: ending tablet copy session 1b16402ace3444f2ba0b2a37688bcf9d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1b16402ace3444f2ba0b2a37688bcf9d
I20260926 23:23:55.937928 7014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:55.941155 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:55.946534 7132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:55.951547 7014 server_base.cc:1034] running on GCE node
W20260926 23:23:55.951890 7134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:55.952406 7014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:55.953151 7014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:55.957810 7014 hybrid_clock.cc:648] HybridClock initialized: now 1790465035957739 us; error 36 us; skew 500 ppm
I20260926 23:23:55.958076 7014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:55 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:55.960491 7014 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 23:23:55.965838 7014 webserver.cc:460] Webserver started at http://127.0.97.61:34793/ using document root <none> and password file <none>
I20260926 23:23:55.966348 7014 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:55.966441 7014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:55.969952 7014 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:55.972240 7141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:55.972752 7014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:23:55.972908 7014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
uuid: "1b16402ace3444f2ba0b2a37688bcf9d"
format_stamp: "Formatted at 2026-09-26 23:23:55 on dist-test-slave-q5g8"
server_key: "1f58d6b13ad832b63b5ac65eaa2d1e55"
server_key_iv: "a7dcedcdb2357631d40df6fe800be281"
server_key_version: "encryptionkey@0"
I20260926 23:23:55.973100 7014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:55.989454 7014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:55.995025 7014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:56.028550 7014 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40969
I20260926 23:23:56.028573 7192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40969 every 8 connection(s)
I20260926 23:23:56.029495 7014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:23:56.036581 7193 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:23:56.039271 7193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Bootstrap starting.
I20260926 23:23:56.050382 7193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:23:56.050896 7193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: Bootstrap complete.
I20260926 23:23:56.051728 7193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } }
I20260926 23:23:56.051954 7193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b16402ace3444f2ba0b2a37688bcf9d, State: Initialized, Role: LEARNER
I20260926 23:23:56.052192 7193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } }
I20260926 23:23:56.053118 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:56.053429 7193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:56.053540 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: LEARNER
I20260926 23:23:56.074383 7207 catalog_manager.cc:1259] Loaded cluster ID: 35e0431d34e74cdfa6af1b055132117a
I20260926 23:23:56.074503 7207 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: loading cluster ID for follower catalog manager: success
I20260926 23:23:56.077283 7207 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: acquiring CA information for follower catalog manager: success
I20260926 23:23:56.079357 7207 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:23:56.091187 7008 heartbeater.cc:503] Master 127.0.97.62:33391 was elected leader, sending a full tablet report...
W20260926 23:23:56.244259 7115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:56.244556 7115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:56.244740 7115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:56.252100 7115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:23:56.252221 7115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:56.252321 7115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:56.252424 7115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:23:56.252507 7115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:23:56.263011 7115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:56.264415 7115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:56.265918 7115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:56.279987 7212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:56.280047 7213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:56.280130 7215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:56.280550 7115 server_base.cc:1034] running on GCE node
I20260926 23:23:56.281558 7115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:56.283151 7115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:56.284382 7115 hybrid_clock.cc:648] HybridClock initialized: now 1790465036284323 us; error 58 us; skew 500 ppm
I20260926 23:23:56.284788 7115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:56.288378 7115 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 23:23:56.289973 7115 webserver.cc:460] Webserver started at http://127.0.97.60:38623/ using document root <none> and password file <none>
I20260926 23:23:56.290649 7115 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:56.290761 7115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:56.291007 7115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:23:56.294116 7115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/instance:
uuid: "4d82dd048cc9427c91bb41ed6305ba2a"
format_stamp: "Formatted at 2026-09-26 23:23:56 on dist-test-slave-q5g8"
server_key: "72acb9aa41066d000c905cf0d91dc783"
server_key_iv: "3def5e754cea3a00ba841ec0743d2890"
server_key_version: "encryptionkey@0"
I20260926 23:23:56.294920 7115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal/instance:
uuid: "4d82dd048cc9427c91bb41ed6305ba2a"
format_stamp: "Formatted at 2026-09-26 23:23:56 on dist-test-slave-q5g8"
server_key: "72acb9aa41066d000c905cf0d91dc783"
server_key_iv: "3def5e754cea3a00ba841ec0743d2890"
server_key_version: "encryptionkey@0"
I20260926 23:23:56.300640 7115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:23:56.304870 7222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:56.306434 7115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:23:56.306622 7115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
uuid: "4d82dd048cc9427c91bb41ed6305ba2a"
format_stamp: "Formatted at 2026-09-26 23:23:56 on dist-test-slave-q5g8"
server_key: "72acb9aa41066d000c905cf0d91dc783"
server_key_iv: "3def5e754cea3a00ba841ec0743d2890"
server_key_version: "encryptionkey@0"
I20260926 23:23:56.306829 7115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:56.325448 7115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:56.330536 7115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:56.330319 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:56.330714 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:56.330963 7115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:56.347944 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.330494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.0.1:51078)
0926 23:23:56.337921 (+ 7427us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.337946 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.338549 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.339927 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.339931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.339963 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:56.340177 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.340197 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.342338 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.342344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.344566 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.344587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.346981 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.346990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.347215 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.347297 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.347428 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.347578 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7253,"thread_start_us":7268,"threads_started":1}
I20260926 23:23:56.347944 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.331159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.0.97.61:40969)
0926 23:23:56.337232 (+ 6073us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.337240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.338058 (+ 818us) server_negotiation.cc:407] Connection header received
0926 23:23:56.338767 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.338772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.338891 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.339005 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:56.340400 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.342160 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.344857 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.346737 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.346938 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.347264 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.347497 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.347626 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5868,"thread_start_us":7517,"threads_started":1}
I20260926 23:23:56.349367 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fec46c64854df8b05a61a9b3831283. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:56.349678 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: LEARNER
I20260926 23:23:56.350740 7112 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: attempting to promote NON_VOTER 1b16402ace3444f2ba0b2a37688bcf9d to VOTER
I20260926 23:23:56.351805 7112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } }
I20260926 23:23:56.353926 7168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 LEARNER]: Refusing update from remote peer 62fec46c64854df8b05a61a9b3831283: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 23:23:56.354526 7227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 23:23:56.358248 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b16402ace3444f2ba0b2a37688bcf9d (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } }
I20260926 23:23:56.359249 7168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1b16402ace3444f2ba0b2a37688bcf9d (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } }
I20260926 23:23:56.362991 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } }
I20260926 23:23:56.363319 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:56.363667 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } }
I20260926 23:23:56.363891 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:56.364125 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } }
I20260926 23:23:56.364403 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:23:56.365542 7115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35185
I20260926 23:23:56.365549 7279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35185 every 8 connection(s)
I20260926 23:23:56.367305 7115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:23:56.368816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 7115
I20260926 23:23:56.373219 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:56.383891 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:23:56.403184 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.370958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.0.97.60:35185)
0926 23:23:56.377743 (+ 6785us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.377761 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.377818 (+ 57us) server_negotiation.cc:407] Connection header received
0926 23:23:56.380528 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.380563 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.381232 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.381765 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.383176 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.385247 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.386567 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.387525 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.387763 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.393687 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.393704 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.393722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.393787 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.397643 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.398568 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.398581 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.398593 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.398853 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.399389 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.399397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.399401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.400355 (+ 954us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.400662 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.401206 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.401709 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6507,"thread_start_us":6279,"threads_started":1}
I20260926 23:23:56.404910 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.387152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.0.97.62:33391)
0926 23:23:56.387351 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.387357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.388136 (+ 779us) server_negotiation.cc:407] Connection header received
0926 23:23:56.388388 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.388396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.388489 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.388599 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.390606 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.391373 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.393266 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.394000 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.394174 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.398340 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.398345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.398349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.398385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.401944 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.402685 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.402690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.402693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.402845 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.403296 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.403301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.403304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.403668 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.403813 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.404348 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.404577 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:23:56.404969 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.387020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:42506)
0926 23:23:56.387977 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.388028 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.388197 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.388721 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.388729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.389609 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.390423 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.390439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.391539 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.391546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.393097 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.393113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.394250 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.394255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.394406 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.395155 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.395188 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.398108 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.402079 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.402087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.402093 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.402543 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.402957 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.402960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.402962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.403092 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.403809 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.403815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.403983 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.404139 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.404486 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":537,"threads_started":1}
I20260926 23:23:56.407191 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:23:56.421595 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.407867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.0.1:51094)
0926 23:23:56.408065 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.408089 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.408182 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.408462 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.408466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.408882 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.409064 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.409091 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.410122 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.410129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.411757 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.411768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.412576 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.412581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.412807 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.413370 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.413390 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.415803 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.419019 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.419024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.419029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.419448 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.419847 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.419852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.419856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.420001 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.420528 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.420533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.420743 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.420909 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.421237 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:23:56.421604 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.407920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.0.97.61:40969)
0926 23:23:56.408082 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.408086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.408132 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:23:56.408280 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.408283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.408343 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.408424 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.409259 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.409985 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.411908 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.412383 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.412501 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.416020 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.416025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.416028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.416063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.418837 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.419586 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.419596 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.419599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.419731 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.420113 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.420119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.420122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.420405 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.420563 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.420997 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.421273 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:23:56.422930 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 23:23:56.436465 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.423805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.0.1:51094)
0926 23:23:56.424024 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.424041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.424144 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.424410 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.424413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.424862 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.425011 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.425019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.426004 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.426007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.426906 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.426917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.427617 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.427621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.427795 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.428392 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.428410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.430622 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.434124 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.434128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.434131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.434501 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.434919 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.434922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.434925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.435089 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.435559 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.435595 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.435811 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.435965 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.436180 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:23:56.436625 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.423765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.0.97.60:35185)
0926 23:23:56.423935 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.423939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.424092 (+ 153us) server_negotiation.cc:407] Connection header received
0926 23:23:56.424238 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.424241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.424298 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.424372 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.425149 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.425890 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.427033 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.427466 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.427567 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.430809 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.430814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.430817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.430847 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.433974 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.434628 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.434631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.434633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.434764 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.435200 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.435203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.435205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.435462 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.435569 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.436035 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.436313 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":57}
I20260926 23:23:56.441522 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Bootstrap starting.
I20260926 23:23:56.444795 7280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:56.445878 7280 log.cc:824] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:56.448580 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: No bootstrap required, opened a new log
I20260926 23:23:56.456835 7280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } }
I20260926 23:23:56.457340 7280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:56.457536 7280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d82dd048cc9427c91bb41ed6305ba2a, State: Initialized, Role: FOLLOWER
I20260926 23:23:56.458200 7280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } }
I20260926 23:23:56.459276 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } } }
I20260926 23:23:56.459662 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:23:56.460497 7280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:23:56.471462 7297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:23:56.471619 7297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:23:56.475286 7115 master_runner.cc:428] Detected that this master 4d82dd048cc9427c91bb41ed6305ba2a is joining an existing cluster
I20260926 23:23:56.475427 7115 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:35185
I20260926 23:23:56.505779 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.480993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.0.97.62:33391)
0926 23:23:56.481597 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.481606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.485081 (+ 3475us) server_negotiation.cc:407] Connection header received
0926 23:23:56.485816 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.485826 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.485932 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.486021 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.488813 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.490097 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.493677 (+ 3580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.494339 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.494452 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.497319 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.497327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.497332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.497383 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.501420 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.502808 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.502829 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.502836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.503016 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.503602 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.503610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.503615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.504034 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.504233 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.504982 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.505318 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
I20260926 23:23:56.507095 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.480471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:42522)
0926 23:23:56.484931 (+ 4460us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.484958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.485594 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.486295 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.486302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.487209 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.488559 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.488576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.491622 (+ 3046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.491628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.493514 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.493525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.494761 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.494770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.495001 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.495821 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.495844 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.497072 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.501591 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.501598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.501602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.502642 (+ 1040us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.503219 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.503224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.503230 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.503449 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.504328 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.504335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.504582 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.506080 (+ 1498us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.506513 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4265,"thread_start_us":8474,"threads_started":1}
I20260926 23:23:56.509943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 6612
I20260926 23:23:56.510788 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.480446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.0.97.60:35185)
0926 23:23:56.481217 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.481234 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.489300 (+ 8066us) server_negotiation.cc:407] Connection header received
0926 23:23:56.491746 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.491752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.491841 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.491915 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.493672 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.494702 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.496795 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.497574 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.497689 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.500948 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.500956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.500960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.501000 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.507664 (+ 6664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.508373 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.508378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.508380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.508540 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.508981 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.508985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.508987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.509444 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.509614 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.510142 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.510385 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618}
I20260926 23:23:56.511775 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.480503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.0.1:51102)
0926 23:23:56.489175 (+ 8672us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.489197 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.491426 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.492114 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.492119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.492888 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.493266 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.493278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.494908 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.494913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.496606 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.496623 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.498073 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.498083 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.498336 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.499080 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.499103 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.500749 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.507803 (+ 7054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.507806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.507809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.508242 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.508714 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.508719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.508722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.508868 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.509676 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.509680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.509857 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.510950 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.511250 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8309,"mutex_wait_us":32,"thread_start_us":7742,"threads_started":1}
I20260926 23:23:56.526229 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.480744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.0.97.61:40969)
0926 23:23:56.481405 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.481414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.496754 (+ 15340us) server_negotiation.cc:407] Connection header received
0926 23:23:56.496839 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.496845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.496940 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.497057 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.498783 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.499964 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.502568 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.503318 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.503460 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.513374 (+ 9914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.513381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.513390 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.513440 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.518243 (+ 4803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.519382 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.519391 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.519397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.519603 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.520169 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.520177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.520183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.520660 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.520858 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.525595 (+ 4737us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.525870 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516}
I20260926 23:23:56.527402 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.481091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.0.1:51102)
0926 23:23:56.493392 (+ 12301us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.493420 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.493525 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.497285 (+ 3760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.497290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.498105 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.498575 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.498588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.500210 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.500227 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.502304 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.502326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.503916 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.503924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.504120 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.511803 (+ 7683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.511826 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.513098 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.518472 (+ 5374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.518482 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.518488 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.519181 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.519786 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.519794 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.519801 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.520025 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.520964 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.520971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.521286 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.526507 (+ 5221us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.526891 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8369,"mutex_wait_us":5,"thread_start_us":4018,"threads_started":1}
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465033, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 23:23:56.541062 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.518244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.0.1:51116)
0926 23:23:56.518630 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.518650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.518759 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.519608 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.519614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.520120 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.520329 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.520341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.522029 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.522036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.523373 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.523388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.524470 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.524477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.524671 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.529829 (+ 5158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.529856 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.533250 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.537805 (+ 4555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.537812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.537818 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.538470 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.538988 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.538993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.538997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.539150 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.539926 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.539931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.540170 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.540314 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.540631 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260926 23:23:56.542523 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.518989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.0.97.60:35185)
0926 23:23:56.519167 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.519174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.519190 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:23:56.519297 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.519303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.519431 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.520487 (+ 1056us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.520660 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.521849 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.523563 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.524287 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.529234 (+ 4947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.533528 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.533536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.533541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.533609 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.537557 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.538685 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.538693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.538696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.538839 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.539314 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.539318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.539341 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.539768 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.541373 (+ 1605us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.541516 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.541825 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:23:56.544387 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.511856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.0.97.1:39011)
0926 23:23:56.522273 (+ 10417us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.522282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.522313 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:23:56.522383 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.522393 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.522702 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.522889 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.524600 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.526341 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.528454 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.529295 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.530518 (+ 1223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.535706 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.535729 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.535755 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.535810 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.539986 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.540983 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.540992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.540995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.541188 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.541683 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.541690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.541696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.542151 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.543599 (+ 1448us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.543773 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.544070 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10227,"thread_start_us":10042,"threads_started":1}
I20260926 23:23:56.544235 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:35185: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 35185
}
attrs {
promote: true
}
}
dest_uuid: "62fec46c64854df8b05a61a9b3831283"
cas_config_opid_index: 6
I20260926 23:23:56.545104 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } }
I20260926 23:23:56.546839 6735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:56.548267 7168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Refusing update from remote peer 62fec46c64854df8b05a61a9b3831283: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 23:23:56.548348 7254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:56.548979 7112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 23:23:56.551608 7254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 FOLLOWER]: Refusing update from remote peer 62fec46c64854df8b05a61a9b3831283: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 23:23:56.551795 7227 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4d82dd048cc9427c91bb41ed6305ba2a (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:56.552551 7112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:56.552397 7168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4d82dd048cc9427c91bb41ed6305ba2a (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:56.556177 7227 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:35185
I20260926 23:23:56.556504 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:56.556908 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:56.557201 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:56.557476 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:56.559222 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:56.559515 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:23:56.560988 7115 master.cc:537] Master@127.0.97.60:35185 shutting down...
I20260926 23:23:56.562974 7253 raft_consensus.cc:858] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } }
W20260926 23:23:56.563141 7253 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } }
W20260926 23:23:56.590932 7233 tablet.cc:2369] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:56.606958 7253 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1b16402ace3444f2ba0b2a37688bcf9d (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:56.612452 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:56.612926 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 23:23:56.612965 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:56.613358 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 23:23:56.613355 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fec46c64854df8b05a61a9b3831283. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: true } } }
I20260926 23:23:56.613791 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 23:23:56.615423 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:56.615731 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:56.617276 7115 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260926 23:23:56.617323 6537 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 23:23:56.617821 7115 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 23:23:56.618021 7115 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: stopping tablet replica
I20260926 23:23:56.627305 7115 master.cc:552] Master@127.0.97.60:35185 shutdown complete.
I20260926 23:23:56.627485 7115 master_runner.cc:305] Clearing existing system tablet
I20260926 23:23:56.627974 7115 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:56.628158 7115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260926 23:23:56.634584 6537 consensus_peers.cc:487] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 -> Peer 4d82dd048cc9427c91bb41ed6305ba2a (127.0.97.60:35185): Couldn't send request to peer 4d82dd048cc9427c91bb41ed6305ba2a. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260926 23:23:56.635520 7115 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.004s
I20260926 23:23:56.637370 7316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:56.637759 7115 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:56.638032 7115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 23:23:56.638154 7115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
uuid: "4d82dd048cc9427c91bb41ed6305ba2a"
format_stamp: "Formatted at 2026-09-26 23:23:56 on dist-test-slave-q5g8"
server_key: "72acb9aa41066d000c905cf0d91dc783"
server_key_iv: "3def5e754cea3a00ba841ec0743d2890"
server_key_version: "encryptionkey@0"
I20260926 23:23:56.643268 7115 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:56.650812 7115 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 23:23:56.651017 7115 log.cc:1196] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 23:23:56.651575 7115 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Deleting consensus metadata
I20260926 23:23:56.651851 7115 master_runner.cc:315] Copying system tablet from 127.0.97.62:33391
I20260926 23:23:56.655344 7115 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33391
I20260926 23:23:56.673022 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.656350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.0.97.62:33391)
0926 23:23:56.656636 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:56.656643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:56.657060 (+ 417us) server_negotiation.cc:407] Connection header received
0926 23:23:56.657287 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:56.657294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:56.657385 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:56.657507 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:56.658697 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.659939 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.662151 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.662883 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.663021 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.665605 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:56.665612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:56.665617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:56.665687 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:56.669712 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.670691 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.670697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.670700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.670856 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:56.671331 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:56.671335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:56.671338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:56.671730 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:56.671876 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:56.672378 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 23:23:56.672640 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 23:23:56.673161 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:56.656225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:42528)
0926 23:23:56.656895 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:56.656920 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:56.657123 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:56.657565 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:56.657571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:56.658171 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:56.658495 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.658509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.660123 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.660129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.661958 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:56.661977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:56.663067 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:56.663074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:56.663275 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:56.664087 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:56.664113 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:56.665360 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:56.669905 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.669913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.669918 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.670509 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.670982 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:56.670988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:56.670992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:56.671186 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:56.671861 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:56.671866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:56.672077 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 23:23:56.672234 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 23:23:56.672571 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":337,"threads_started":1}
I20260926 23:23:56.674218 6571 tablet_copy_service.cc:140] P 62fec46c64854df8b05a61a9b3831283: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4d82dd048cc9427c91bb41ed6305ba2a ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:42528)
I20260926 23:23:56.674397 6571 tablet_copy_service.cc:161] P 62fec46c64854df8b05a61a9b3831283: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4d82dd048cc9427c91bb41ed6305ba2a at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:42528: session id = 4d82dd048cc9427c91bb41ed6305ba2a-00000000000000000000000000000000
I20260926 23:23:56.678104 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 23:23:56.680291 7115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 23:23:56.688285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 6612
I20260926 23:23:56.688670 7115 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: tablet copy: Starting download of 0 data blocks...
I20260926 23:23:56.689241 7115 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: tablet copy: Starting download of 1 WAL segments...
I20260926 23:23:56.692116 7115 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 23:23:56.697597 6571 tablet_copy_service.cc:342] P 62fec46c64854df8b05a61a9b3831283: Request end of tablet copy session 4d82dd048cc9427c91bb41ed6305ba2a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:42528
I20260926 23:23:56.697777 6571 tablet_copy_service.cc:433] P 62fec46c64854df8b05a61a9b3831283: ending tablet copy session 4d82dd048cc9427c91bb41ed6305ba2a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4d82dd048cc9427c91bb41ed6305ba2a
I20260926 23:23:56.712364 7115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:23:56.713258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39011
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33391
--webserver_port=32775
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:56.715653 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:56.718467 7324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:56.729314 7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:56.729379 7115 server_base.cc:1034] running on GCE node
I20260926 23:23:56.729981 7115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:56.730746 7115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:56.731958 7115 hybrid_clock.cc:648] HybridClock initialized: now 1790465036731911 us; error 40 us; skew 500 ppm
I20260926 23:23:56.732244 7115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:56 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:56.734313 7115 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 23:23:56.741717 7115 webserver.cc:460] Webserver started at http://127.0.97.60:39073/ using document root <none> and password file <none>
I20260926 23:23:56.742055 7115 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:56.742192 7115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:56.744724 7115 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 23:23:56.746439 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:56.746877 7115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 23:23:56.747015 7115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
uuid: "4d82dd048cc9427c91bb41ed6305ba2a"
format_stamp: "Formatted at 2026-09-26 23:23:56 on dist-test-slave-q5g8"
server_key: "72acb9aa41066d000c905cf0d91dc783"
server_key_iv: "3def5e754cea3a00ba841ec0743d2890"
server_key_version: "encryptionkey@0"
I20260926 23:23:56.747207 7115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:56.766932 7115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:56.770690 7115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:56.800736 7115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35185
I20260926 23:23:56.800796 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35185 every 8 connection(s)
I20260926 23:23:56.801718 7115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:23:56.806751 7386 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:23:56.810875 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Bootstrap starting.
I20260926 23:23:56.833328 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:23:56.833911 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Bootstrap complete.
I20260926 23:23:56.835008 7386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } }
I20260926 23:23:56.835364 7386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d82dd048cc9427c91bb41ed6305ba2a, State: Initialized, Role: LEARNER
I20260926 23:23:56.835688 7386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } }
I20260926 23:23:56.837296 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:56.837610 7386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:23:56.837891 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: LEARNER
I20260926 23:23:56.864570 7400 catalog_manager.cc:1259] Loaded cluster ID: 35e0431d34e74cdfa6af1b055132117a
I20260926 23:23:56.864845 7400 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: loading cluster ID for follower catalog manager: success
I20260926 23:23:56.868731 7400 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: acquiring CA information for follower catalog manager: success
I20260926 23:23:56.871835 7400 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:23:57.006528 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:57.007009 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:57.014173 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.007109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51132 (local address 127.0.97.60:35185)
0926 23:23:57.007844 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.007848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.007869 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:23:57.007932 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.007935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.008004 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.008129 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:57.008677 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.009714 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.012042 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.013023 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.013164 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.013374 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:57.013751 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 23:23:57.013864 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":476,"threads_started":1}
I20260926 23:23:57.014357 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.006948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.0.1:51132)
0926 23:23:57.007272 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:57.007295 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:57.007437 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:57.008196 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:57.008205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:57.008252 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:57.008505 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.008519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.009879 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.009884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.011843 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.011865 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.013236 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.013246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.013537 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.013599 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 23:23:57.013724 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 23:23:57.013942 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260926 23:23:57.015265 7361 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260926 23:23:57.015764 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fec46c64854df8b05a61a9b3831283. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: true } } }
I20260926 23:23:57.016084 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: LEARNER
I20260926 23:23:57.016754 7227 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: attempting to promote NON_VOTER 4d82dd048cc9427c91bb41ed6305ba2a to VOTER
I20260926 23:23:57.017475 7227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } }
I20260926 23:23:57.019688 7361 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 LEARNER]: Refusing update from remote peer 62fec46c64854df8b05a61a9b3831283: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 23:23:57.019759 7168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Refusing update from remote peer 62fec46c64854df8b05a61a9b3831283: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 23:23:57.020326 7227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 23:23:57.020620 7228 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 23:23:57.023196 7228 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4d82dd048cc9427c91bb41ed6305ba2a (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.028218 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.028508 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:57.028178 7168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4d82dd048cc9427c91bb41ed6305ba2a (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.028935 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.029124 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:57.029407 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.029630 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283 [sys.catalog]: This master's current role is: LEADER
I20260926 23:23:57.029477 7361 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4d82dd048cc9427c91bb41ed6305ba2a (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.032470 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.032786 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:23:57.033526 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62fec46c64854df8b05a61a9b3831283" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:23:57.033818 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: FOLLOWER
W20260926 23:23:57.142256 7326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:57.142566 7326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:57.142728 7326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:57.150718 7326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:57.150851 7326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:57.151048 7326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:23:57.163614 7326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=32775
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:57.165231 7326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:57.166930 7326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:57.183341 7407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:57.183390 7408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:57.183295 7410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:57.184966 7326 server_base.cc:1034] running on GCE node
I20260926 23:23:57.185808 7326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:57.187084 7326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:57.188356 7326 hybrid_clock.cc:648] HybridClock initialized: now 1790465037188284 us; error 81 us; skew 500 ppm
I20260926 23:23:57.188813 7326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:57.192384 7326 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:23:57.194242 7326 webserver.cc:460] Webserver started at http://127.0.97.1:32775/ using document root <none> and password file <none>
I20260926 23:23:57.195009 7326 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:57.195169 7326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:57.201294 7326 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 23:23:57.204403 7417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:57.206089 7326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:23:57.206306 7326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "070ebf88cc1b44e49719cc523a51e250"
format_stamp: "Formatted at 2026-09-26 23:23:53 on dist-test-slave-q5g8"
server_key: "ea3c7ce6fc4fc493e680f9231d2671f3"
server_key_iv: "63380ed86a193ca4513b4f0b93860cb7"
server_key_version: "encryptionkey@0"
I20260926 23:23:57.207377 7326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:57.231732 7326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:57.236544 7326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:57.236879 7326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:57.238147 7326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:57.239647 7326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:57.239735 7326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:57.239864 7326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:57.239965 7326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:57.279330 7326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39011
I20260926 23:23:57.279390 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39011 every 8 connection(s)
I20260926 23:23:57.280915 7326 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:23:57.284054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 7326
I20260926 23:23:57.284408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 6746
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:23:57.336314 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.292051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35253 (local address 127.0.97.60:35185)
0926 23:23:57.292313 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.292323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.310203 (+ 17880us) server_negotiation.cc:407] Connection header received
0926 23:23:57.311641 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.311648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.311729 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.311796 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.314186 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.315228 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.319596 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.320236 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.320348 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.327878 (+ 7530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.327887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.327892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.327944 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:57.331918 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.333085 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.333092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.333097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.333302 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.333907 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.333913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.333918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.334346 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:57.334507 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:57.335648 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0926 23:23:57.335929 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:23:57.337954 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.285746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.97.2:37491)
0926 23:23:57.309082 (+ 23336us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.309090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.309150 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:23:57.309228 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.309237 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.309551 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.310391 (+ 840us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.312873 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.314514 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.316386 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.317557 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.317701 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.328268 (+ 10567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.328288 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.328309 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.328359 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:57.334070 (+ 5711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.335115 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.335126 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.335130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.335297 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.335765 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.335771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.335774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.336200 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:57.336485 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:57.337138 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 23:23:57.337485 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23159,"thread_start_us":23017,"threads_started":1}
I20260926 23:23:57.339077 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.291492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.97.1:35253)
0926 23:23:57.310089 (+ 18597us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:57.310113 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:57.311457 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:57.312116 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:57.312121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:57.313654 (+ 1533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:57.313987 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.314002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.315444 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.315450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.317791 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.317809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.320744 (+ 2935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.320753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.321065 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.322135 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:57.322183 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:57.327608 (+ 5425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:57.332134 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.332162 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.332185 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.332916 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.333479 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.333483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.333487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.333753 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.334607 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:57.334617 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:57.335126 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 23:23:57.336558 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0926 23:23:57.337218 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18460,"mutex_wait_us":47,"thread_start_us":14730,"threads_started":1}
I20260926 23:23:57.339589 6869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:57.340890 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.284636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34449 (local address 127.0.97.61:40969)
0926 23:23:57.307779 (+ 23143us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.307788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.307816 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:23:57.309675 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.309682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.309806 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.309912 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.315870 (+ 5958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.317030 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.319034 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.319706 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.319801 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.328383 (+ 8582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.328391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.328396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.328445 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:57.333353 (+ 4908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.337392 (+ 4039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.337398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.337401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.337776 (+ 375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.338555 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.338560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.338563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.338945 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:57.339089 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:57.340239 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0926 23:23:57.340487 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22943,"thread_start_us":22767,"threads_started":1}
I20260926 23:23:57.340996 7530 heartbeater.cc:347] Connected to a master server at 127.0.97.60:35185
I20260926 23:23:57.341315 7530 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:57.341693 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.285421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.1:53471)
0926 23:23:57.306074 (+ 20653us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:57.306100 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:57.309154 (+ 3054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:57.310390 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:57.310404 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:57.311458 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:57.313921 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.313933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.315482 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.315488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.319384 (+ 3896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.319399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.321167 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.321175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.321360 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.323490 (+ 2130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:57.323512 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:57.327224 (+ 3712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:57.337708 (+ 10484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.337715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.337719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.338316 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.338814 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.338821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.338827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.338998 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.339747 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:57.339752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:57.340392 (+ 640us) client_negotiation.cc:769] Sending connection context
0926 23:23:57.340582 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 23:23:57.341067 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20486,"mutex_wait_us":1,"thread_start_us":19515,"threads_started":1}
I20260926 23:23:57.342293 7530 heartbeater.cc:511] Master 127.0.97.60:35185 requested a full tablet report, sending...
I20260926 23:23:57.342526 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.285006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53471 (local address 127.0.97.62:33391)
0926 23:23:57.308195 (+ 23189us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.308204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.308230 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:23:57.309649 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.309656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.309758 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.309879 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.314108 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.315319 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.320018 (+ 4699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.320758 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.320878 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.329597 (+ 8719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.329606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.329611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.329668 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:57.335802 (+ 6134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.338499 (+ 2697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.338505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.338508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.338677 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.339179 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.339185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.339188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.339593 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:57.341458 (+ 1865us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:57.341664 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 23:23:57.341991 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23034,"thread_start_us":22944,"threads_started":1}
I20260926 23:23:57.343815 7531 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33391
I20260926 23:23:57.343935 7531 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:57.344424 7531 heartbeater.cc:511] Master 127.0.97.62:33391 requested a full tablet report, sending...
I20260926 23:23:57.345203 7351 ts_manager.cc:194] Registered new tserver with Master: 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011)
I20260926 23:23:57.345623 6551 ts_manager.cc:194] Re-registered known tserver with Master: 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011)
I20260926 23:23:57.346565 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53471
I20260926 23:23:57.345077 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.285296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.97.1:34449)
0926 23:23:57.305438 (+ 20142us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:57.305498 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:57.309140 (+ 3642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:57.310392 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:57.310404 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:57.312469 (+ 2065us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:57.313668 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.313698 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.317255 (+ 3557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.317262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.318847 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.318861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.322197 (+ 3336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.322204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.322380 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.324932 (+ 2552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:57.324956 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:57.328112 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:57.335948 (+ 7836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.335952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.335956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.337225 (+ 1269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.337953 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.337958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.337962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.338419 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.339461 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:57.339467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:57.339962 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 23:23:57.340120 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 23:23:57.344387 (+ 4267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19758,"thread_start_us":24371,"threads_started":1}
I20260926 23:23:57.350905 7532 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40969
I20260926 23:23:57.351121 7532 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:57.351647 7532 heartbeater.cc:511] Master 127.0.97.61:40969 requested a full tablet report, sending...
I20260926 23:23:57.353618 7158 ts_manager.cc:194] Registered new tserver with Master: 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011)
I20260926 23:23:57.435366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 6746
I20260926 23:23:57.451850 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37491
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33391
--webserver_port=43549
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:23:57.802523 7542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:57.802917 7542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:57.803084 7542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:57.814916 7542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:57.815079 7542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:57.815290 7542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:23:57.825789 7542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43549
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:57.827145 7542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:57.828440 7542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:57.842186 7549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:57.842160 7548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:57.842741 7551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:57.843461 7542 server_base.cc:1034] running on GCE node
I20260926 23:23:57.844123 7542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:57.845197 7542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:57.846427 7542 hybrid_clock.cc:648] HybridClock initialized: now 1790465037846378 us; error 44 us; skew 500 ppm
I20260926 23:23:57.846778 7542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:57.850198 7542 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:23:57.851648 7542 webserver.cc:460] Webserver started at http://127.0.97.2:43549/ using document root <none> and password file <none>
I20260926 23:23:57.852370 7542 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:57.852556 7542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:57.858361 7542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:23:57.861179 7558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:57.862632 7542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:23:57.862838 7542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "68a492260d0b4fb881a25623fde1b624"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "27693b6600a001fe9631faaf90112eb0"
server_key_iv: "aabfae9336a36c6c17a3f6b535ce212f"
server_key_version: "encryptionkey@0"
I20260926 23:23:57.863799 7542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:57.887697 7542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:57.891760 7542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:57.892004 7542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:57.893013 7542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:57.894416 7542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:57.894506 7542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:57.894582 7542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:57.894634 7542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:57.932226 7542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37491
I20260926 23:23:57.932303 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37491 every 8 connection(s)
I20260926 23:23:57.934978 7542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:23:57.944763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 7542
I20260926 23:23:57.945101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 6880
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 23:23:57 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:23:57.996902 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.940524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38235 (local address 127.0.97.60:35185)
0926 23:23:57.969684 (+ 29160us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.969693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.969722 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:23:57.971341 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.971350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.971480 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.971612 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.976972 (+ 5360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.978223 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.982272 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.982772 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.982863 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.989074 (+ 6211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.989084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.989089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.989152 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:57.992675 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.993747 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.993755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.993763 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.993912 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.994555 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.994562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.994565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.994953 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:57.995083 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:57.996207 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0926 23:23:57.996458 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29003,"thread_start_us":28877,"threads_started":1}
I20260926 23:23:57.998301 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.939549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48205 (local address 127.0.97.61:40969)
0926 23:23:57.940110 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.940118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.976098 (+ 35980us) server_negotiation.cc:407] Connection header received
0926 23:23:57.976299 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.976305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.976396 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.976502 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.977788 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.978757 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.982262 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.982990 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.983111 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.990319 (+ 7208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.990326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.990330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.990377 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:57.993920 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.994792 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.994798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.994800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.994952 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.996108 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.996113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.996116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.996458 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:57.996599 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:57.997712 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0926 23:23:57.997924 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":2412,"threads_started":1}
I20260926 23:23:57.999590 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.940427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.97.2:38235)
0926 23:23:57.966762 (+ 26335us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:57.966792 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:57.970712 (+ 3920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:57.972858 (+ 2146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:57.972879 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:57.975879 (+ 3000us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:57.976648 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.976670 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.979510 (+ 2840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.979518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.982060 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.982077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.983308 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.983317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.983836 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.985387 (+ 1551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:57.985411 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:57.988704 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:57.992907 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.992920 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.992945 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.993585 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.994066 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.994071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.994075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.994406 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.995183 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:57.995195 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:57.995717 (+ 522us) client_negotiation.cc:769] Sending connection context
0926 23:23:57.997160 (+ 1443us) client_negotiation.cc:240] Negotiation successful
0926 23:23:57.997767 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26030,"mutex_wait_us":656,"thread_start_us":35158,"threads_started":1}
I20260926 23:23:58.001132 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.940347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.2:60435)
0926 23:23:57.965918 (+ 25571us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:57.966408 (+ 490us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:57.972272 (+ 5864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:57.972919 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:57.972927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:57.976415 (+ 3488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:57.976887 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.976899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.978455 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.978463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.982059 (+ 3596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.982077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.983308 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.983317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.983833 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.985016 (+ 1183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:57.985059 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:57.990221 (+ 5162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:57.995420 (+ 5199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.995424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.995427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.995960 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.996672 (+ 712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.996832 (+ 160us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.996838 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.997000 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.999593 (+ 2593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:57.999599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:57.999758 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 23:23:57.999930 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 23:23:58.000247 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24187,"mutex_wait_us":240,"spinlock_wait_cycles":164736,"thread_start_us":21994,"threads_started":1}
I20260926 23:23:58.001613 7671 heartbeater.cc:347] Connected to a master server at 127.0.97.60:35185
I20260926 23:23:58.001673 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.941710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.97.2:48205)
0926 23:23:57.975983 (+ 34273us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:57.976007 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:57.976170 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:57.976710 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:57.976721 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:57.977330 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:57.977594 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.977605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.978875 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.978881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.982024 (+ 3143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:57.982046 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.983493 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.983501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:57.983818 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.985016 (+ 1198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:57.985059 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:57.990060 (+ 5001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:57.994068 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.994072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.994075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.994655 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.995096 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:57.995103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:57.995106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:57.995283 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:57.997134 (+ 1851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:57.997138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:57.997271 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 23:23:57.997414 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:23:57.998723 (+ 1309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34112,"mutex_wait_us":1150,"spinlock_wait_cycles":31104,"thread_start_us":23957,"threads_started":1}
I20260926 23:23:58.002005 7671 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:58.002609 7673 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40969
I20260926 23:23:58.002753 7673 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:58.003077 7671 heartbeater.cc:511] Master 127.0.97.60:35185 requested a full tablet report, sending...
I20260926 23:23:58.003276 7673 heartbeater.cc:511] Master 127.0.97.61:40969 requested a full tablet report, sending...
I20260926 23:23:58.005087 7158 ts_manager.cc:194] Registered new tserver with Master: 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491)
I20260926 23:23:58.005247 7351 ts_manager.cc:194] Registered new tserver with Master: 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491)
I20260926 23:23:58.006011 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.941571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60435 (local address 127.0.97.62:33391)
0926 23:23:57.954880 (+ 13309us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.954891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.966638 (+ 11747us) server_negotiation.cc:407] Connection header received
0926 23:23:57.972453 (+ 5815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.972461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.972556 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.972671 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.977110 (+ 4439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.978213 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.982235 (+ 4022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.982991 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.983130 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.990458 (+ 7328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.990468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.990474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.990521 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:57.995244 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.996087 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.996096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.996101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.996282 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:57.998877 (+ 2595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:57.998884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:57.998890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:57.999333 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:57.999659 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.000897 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.002932 (+ 2035us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13121,"thread_start_us":12982,"threads_started":1}
I20260926 23:23:58.007091 7672 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33391
I20260926 23:23:58.007251 7672 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:58.007761 7672 heartbeater.cc:511] Master 127.0.97.62:33391 requested a full tablet report, sending...
I20260926 23:23:58.009044 6551 ts_manager.cc:194] Re-registered known tserver with Master: 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491)
I20260926 23:23:58.009948 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60435
I20260926 23:23:58.011233 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:57.946869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.97.3:41405)
0926 23:23:57.956138 (+ 9269us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:57.956151 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:57.957799 (+ 1648us) server_negotiation.cc:407] Connection header received
0926 23:23:57.957901 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:57.957914 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:57.958271 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:57.958481 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:57.965526 (+ 7045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.970073 (+ 4547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.973718 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:57.974753 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:57.974905 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:57.996599 (+ 21694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:57.996625 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:57.996648 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:57.996742 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:58.005580 (+ 8838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.007675 (+ 2095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.007684 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.007687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.007895 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.008464 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.008473 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.008479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.009116 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:58.009472 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.010272 (+ 800us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.010615 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9022,"thread_start_us":8945,"threads_started":1}
I20260926 23:23:58.013096 7003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:58.108202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 6880
I20260926 23:23:58.124938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41405
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33391
--webserver_port=43161
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:23:58.349785 7531 heartbeater.cc:503] Master 127.0.97.62:33391 was elected leader, sending a full tablet report...
W20260926 23:23:58.478799 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:23:58.479182 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:23:58.479372 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:23:58.487087 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:23:58.487253 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:23:58.487450 7684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:23:58.498212 7684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43161
--tserver_master_addrs=127.0.97.62:33391,127.0.97.61:40969,127.0.97.60:35185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:23:58.499890 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:23:58.501817 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:23:58.517374 7690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:58.517314 7689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:23:58.518002 7692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:23:58.518638 7684 server_base.cc:1034] running on GCE node
I20260926 23:23:58.519354 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:23:58.520952 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:23:58.522202 7684 hybrid_clock.cc:648] HybridClock initialized: now 1790465038522117 us; error 77 us; skew 500 ppm
I20260926 23:23:58.522639 7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:23:58 dist-test-slave-q5g8 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465038, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:23:58.526595 7684 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:23:58.528264 7684 webserver.cc:460] Webserver started at http://127.0.97.3:43161/ using document root <none> and password file <none>
I20260926 23:23:58.529177 7684 fs_manager.cc:359] Metadata directory not provided
I20260926 23:23:58.529356 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:23:58.535509 7684 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:23:58.539494 7699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:58.541710 7684 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:23:58.541906 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "2fccafc1129343ae9e66789b30ee0f66"
format_stamp: "Formatted at 2026-09-26 23:23:54 on dist-test-slave-q5g8"
server_key: "15385fef2004bdfdb0a3457a4f9be8ac"
server_key_iv: "b510c2e4c980b745b3b6793ae7cf97c3"
server_key_version: "encryptionkey@0"
I20260926 23:23:58.543026 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:23:58.565660 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:23:58.570493 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:23:58.570817 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:23:58.571908 7684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:23:58.573484 7684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:23:58.573589 7684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:58.573719 7684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:23:58.573803 7684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:23:58.607520 7684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41405
I20260926 23:23:58.607569 7811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41405 every 8 connection(s)
I20260926 23:23:58.609198 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:23:58.618086 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 7684
Sep 26 23:23:58 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465038, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 23:23:58 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465038, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 23:23:58 dist-test-slave-q5g8 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465038, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:23:58.674695 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.651085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.0.97.61:40969)
0926 23:23:58.651767 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.651776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.651801 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:23:58.651864 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.651872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.651983 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.652124 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:58.656179 (+ 4055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.657471 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.659553 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.660697 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.660817 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.664925 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:58.664938 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:58.664945 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:58.664986 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:58.670015 (+ 5029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.670972 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.670981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.670988 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.671163 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.671671 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.671680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.671690 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.672087 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:58.673822 (+ 1735us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.674041 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.674287 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":549,"threads_started":1}
I20260926 23:23:58.681836 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.627331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.0.97.60:35185)
0926 23:23:58.649507 (+ 22176us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.649515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.652421 (+ 2906us) server_negotiation.cc:407] Connection header received
0926 23:23:58.652485 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.652490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.652576 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.652765 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:58.656802 (+ 4037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.658080 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.660719 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.663801 (+ 3082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.664994 (+ 1193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.668944 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:58.668952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:58.668960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:58.669013 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:58.678195 (+ 9182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.679083 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.679089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.679092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.679267 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.679731 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.679739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.679745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.680052 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:58.680319 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.681260 (+ 941us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.681464 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22021,"thread_start_us":24195,"threads_started":1}
I20260926 23:23:58.685113 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.613275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43163 (local address 127.0.97.60:35185)
0926 23:23:58.643568 (+ 30293us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.643584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.643621 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:23:58.646864 (+ 3243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.646874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.647069 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.647217 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:58.657333 (+ 10116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.658540 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.665262 (+ 6722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.665997 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.666123 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.677470 (+ 11347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:58.677478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:58.677483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:58.677526 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:58.681493 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.682245 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.682250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.682253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.682419 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.682894 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.682899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.682901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.683259 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:58.683368 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.684431 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.684651 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30101,"thread_start_us":30381,"threads_started":1}
I20260926 23:23:58.685488 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.612653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55313 (local address 127.0.97.62:33391)
0926 23:23:58.655257 (+ 42604us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.655263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.655279 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:23:58.655335 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.655341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.655421 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.655544 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:58.656654 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.657768 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.662108 (+ 4340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.662771 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.662901 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.670732 (+ 7831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:58.670740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:58.670744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:58.670788 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:58.674846 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.676725 (+ 1879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.676732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.676736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.676894 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.683134 (+ 6240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.683139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.683143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.683503 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:58.683637 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.684903 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.685146 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42480,"thread_start_us":42340,"threads_started":1}
I20260926 23:23:58.685854 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.612665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35357 (local address 127.0.97.61:40969)
0926 23:23:58.649713 (+ 37048us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.649721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.650464 (+ 743us) server_negotiation.cc:407] Connection header received
0926 23:23:58.650611 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.650616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.650719 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.650821 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:58.653751 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.654992 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.664273 (+ 9281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.664995 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.665490 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.676585 (+ 11095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:58.676592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:58.676597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:58.676635 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:58.682172 (+ 5537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.683301 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.683306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.683309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.683465 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.683840 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.683847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.683850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.684192 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:58.684338 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.684773 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.684991 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36887,"thread_start_us":38183,"threads_started":1}
I20260926 23:23:58.686491 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.613825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:55313)
0926 23:23:58.648507 (+ 34682us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:58.648534 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:58.648724 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:58.655557 (+ 6833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:58.655561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:58.656103 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:58.656470 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.656485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.658747 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.658754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.661000 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.661013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.664437 (+ 3424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.664444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.664788 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.666586 (+ 1798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:58.666628 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:58.670460 (+ 3832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:58.675050 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:58.675064 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:58.675089 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:58.676329 (+ 1240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:58.678431 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:58.678437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:58.678442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:58.682566 (+ 4124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:58.683866 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:58.683871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:58.684017 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 23:23:58.684314 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 23:23:58.684922 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34482,"spinlock_wait_cycles":36992,"thread_start_us":35243,"threads_started":1}
I20260926 23:23:58.686597 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.613624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.97.3:35357)
0926 23:23:58.650192 (+ 36568us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:58.650211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:58.650327 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:58.651332 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:58.651345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:58.652487 (+ 1142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:58.653515 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.653546 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.662630 (+ 9084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.662636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.664126 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.664138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.670515 (+ 6377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.670523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.670759 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.671464 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:58.671488 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:58.675556 (+ 4068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:58.682566 (+ 7010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:58.682571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:58.682574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:58.683163 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:58.683595 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:58.683601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:58.683605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:58.683722 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:58.684304 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:58.684310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:58.684443 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 23:23:58.684592 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 23:23:58.685862 (+ 1270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36386,"mutex_wait_us":1509,"spinlock_wait_cycles":1408,"thread_start_us":32619,"threads_started":1}
I20260926 23:23:58.686599 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.613624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.97.3:43163)
0926 23:23:58.642691 (+ 29067us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:58.642779 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:58.646304 (+ 3525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:58.655239 (+ 8935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:58.655245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:58.656884 (+ 1639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:23:58.657134 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.657144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.661148 (+ 4004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.661151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.662463 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.662478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.668624 (+ 6146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.668631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.668950 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.670791 (+ 1841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:23:58.670814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:23:58.676651 (+ 5837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:23:58.681637 (+ 4986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:58.681642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:58.681648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:58.682116 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:58.682573 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:23:58.682578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:23:58.682581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:23:58.682764 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:23:58.683476 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:23:58.683486 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:23:58.683972 (+ 486us) client_negotiation.cc:769] Sending connection context
0926 23:23:58.685350 (+ 1378us) client_negotiation.cc:240] Negotiation successful
0926 23:23:58.685644 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28561,"thread_start_us":36087,"threads_started":1}
I20260926 23:23:58.688195 7812 heartbeater.cc:347] Connected to a master server at 127.0.97.60:35185
I20260926 23:23:58.688244 7814 heartbeater.cc:347] Connected to a master server at 127.0.97.61:40969
I20260926 23:23:58.688304 7813 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33391
I20260926 23:23:58.688596 7814 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:58.688593 7812 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:58.688594 7813 heartbeater.cc:464] Registering TS with master...
I20260926 23:23:58.689801 7812 heartbeater.cc:511] Master 127.0.97.60:35185 requested a full tablet report, sending...
I20260926 23:23:58.689867 7813 heartbeater.cc:511] Master 127.0.97.62:33391 requested a full tablet report, sending...
I20260926 23:23:58.689903 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.655203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.0.97.62:33391)
0926 23:23:58.655828 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.655835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.655854 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:23:58.655901 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.655905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.655975 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.656074 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:58.660830 (+ 4756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.661942 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.663943 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.664608 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.664765 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.675977 (+ 11212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:58.675984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:58.675989 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:58.676029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:58.679696 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.680532 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.680537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.680540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.680726 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:58.683691 (+ 2965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:58.683695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:58.683698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:58.683989 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:58.684219 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.689430 (+ 5211us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.689623 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":328,"threads_started":1}
I20260926 23:23:58.690766 7814 heartbeater.cc:511] Master 127.0.97.61:40969 requested a full tablet report, sending...
I20260926 23:23:58.692076 6551 ts_manager.cc:194] Re-registered known tserver with Master: 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405)
I20260926 23:23:58.692237 7351 ts_manager.cc:194] Registered new tserver with Master: 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405)
I20260926 23:23:58.692517 7158 ts_manager.cc:194] Registered new tserver with Master: 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405)
I20260926 23:23:58.693121 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55313
I20260926 23:23:58.694540 6551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:23:58.702023 6551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:23:58.736799 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.722810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.0.97.3:41405)
0926 23:23:58.724206 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.724220 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.725177 (+ 957us) server_negotiation.cc:407] Connection header received
0926 23:23:58.725325 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.725337 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.725697 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.725945 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:58.727194 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.729395 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.732974 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.735190 (+ 2216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.735340 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.735611 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.736255 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.736423 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":911,"threads_started":1}
I20260926 23:23:58.736749 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.722240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41405 (local address 127.0.0.1:44920)
0926 23:23:58.724807 (+ 2567us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:58.724832 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:58.725003 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:58.726336 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:58.726344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:58.726380 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:58.726903 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.726918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.729636 (+ 2718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.729646 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.732723 (+ 3077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.732747 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.735821 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.735830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.736032 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.736057 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:23:58.736190 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 23:23:58.736351 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2372,"thread_start_us":13273,"threads_started":1}
I20260926 23:23:58.741297 7747 tablet_service.cc:1467] Processing CreateTablet for tablet 75f3b024717f4d03a482100c4ba6e944 (DEFAULT_TABLE table=test-table [id=de17cc34e89748c696b769d19da1c8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:58.742727 7747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f3b024717f4d03a482100c4ba6e944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:58.746555 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.722795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37491 (local address 127.0.0.1:43342)
0926 23:23:58.734819 (+ 12024us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:58.734853 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:58.735030 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:58.735837 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:58.735843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:58.735884 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:58.736345 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.736359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.740069 (+ 3710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.740080 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.743181 (+ 3101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.743205 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.745315 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.745326 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.745595 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.745657 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 23:23:58.745748 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 23:23:58.746061 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11803,"mutex_wait_us":105,"thread_start_us":12357,"threads_started":1}
I20260926 23:23:58.747120 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.723661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.0.97.2:37491)
0926 23:23:58.724383 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.724394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.734986 (+ 10592us) server_negotiation.cc:407] Connection header received
0926 23:23:58.735206 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.735215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.735561 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.736596 (+ 1035us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:58.737446 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.739791 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.743346 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.745026 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.745939 (+ 913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.746214 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.746295 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.746577 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":811,"threads_started":1}
I20260926 23:23:58.750591 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.724644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.0.97.1:39011)
0926 23:23:58.742094 (+ 17450us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:58.742105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:58.742147 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:23:58.742278 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:58.742288 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:58.742586 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:58.742823 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:58.743690 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.745252 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.747880 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.749371 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.749556 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.749797 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:58.750061 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 23:23:58.750243 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17232,"thread_start_us":18311,"threads_started":1}
I20260926 23:23:58.750607 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:58.724116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39011 (local address 127.0.0.1:55834)
0926 23:23:58.735850 (+ 11734us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:58.736402 (+ 552us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:58.736521 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:58.742960 (+ 6439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:58.742965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:58.743001 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:58.743431 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.743444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.745417 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.745424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.747623 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:58.747640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:58.749608 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:58.749616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:58.749825 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:58.749867 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:23:58.750017 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 23:23:58.750175 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11541,"thread_start_us":1061,"threads_started":1}
I20260926 23:23:58.751605 7606 tablet_service.cc:1467] Processing CreateTablet for tablet 75f3b024717f4d03a482100c4ba6e944 (DEFAULT_TABLE table=test-table [id=de17cc34e89748c696b769d19da1c8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:58.753708 7606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f3b024717f4d03a482100c4ba6e944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:58.754649 7465 tablet_service.cc:1467] Processing CreateTablet for tablet 75f3b024717f4d03a482100c4ba6e944 (DEFAULT_TABLE table=test-table [id=de17cc34e89748c696b769d19da1c8b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:23:58.756177 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f3b024717f4d03a482100c4ba6e944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:23:58.762482 7849 tablet_bootstrap.cc:492] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Bootstrap starting.
I20260926 23:23:58.767529 7849 tablet_bootstrap.cc:654] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:58.769613 7849 log.cc:824] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:58.770042 7850 tablet_bootstrap.cc:492] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Bootstrap starting.
I20260926 23:23:58.773108 7850 tablet_bootstrap.cc:654] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:58.774796 7850 log.cc:824] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:58.776443 7852 tablet_bootstrap.cc:492] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Bootstrap starting.
I20260926 23:23:58.777964 7849 tablet_bootstrap.cc:492] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: No bootstrap required, opened a new log
I20260926 23:23:58.778110 7850 tablet_bootstrap.cc:492] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: No bootstrap required, opened a new log
I20260926 23:23:58.778388 7850 ts_tablet_manager.cc:1397] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:23:58.778398 7849 ts_tablet_manager.cc:1397] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260926 23:23:58.780220 7852 tablet_bootstrap.cc:654] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Neither blocks nor log segments found. Creating new log.
I20260926 23:23:58.781582 7852 log.cc:824] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Log is configured to *not* fsync() on all Append() calls
I20260926 23:23:58.785563 7852 tablet_bootstrap.cc:492] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: No bootstrap required, opened a new log
I20260926 23:23:58.785854 7852 ts_tablet_manager.cc:1397] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 23:23:58.786890 7849 raft_consensus.cc:348] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:58.787305 7849 raft_consensus.cc:374] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:58.787418 7849 raft_consensus.cc:729] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fccafc1129343ae9e66789b30ee0f66, State: Initialized, Role: FOLLOWER
I20260926 23:23:58.788259 7849 consensus_queue.cc:260] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:58.789309 7850 raft_consensus.cc:348] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:58.789759 7850 raft_consensus.cc:374] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:58.789880 7850 raft_consensus.cc:729] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a492260d0b4fb881a25623fde1b624, State: Initialized, Role: FOLLOWER
I20260926 23:23:58.790527 7850 consensus_queue.cc:260] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:58.792858 7850 ts_tablet_manager.cc:1428] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260926 23:23:58.793905 7672 heartbeater.cc:503] Master 127.0.97.62:33391 was elected leader, sending a full tablet report...
I20260926 23:23:58.797210 7852 raft_consensus.cc:348] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:58.797693 7852 raft_consensus.cc:374] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:23:58.797821 7852 raft_consensus.cc:729] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070ebf88cc1b44e49719cc523a51e250, State: Initialized, Role: FOLLOWER
I20260926 23:23:58.798583 7852 consensus_queue.cc:260] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:58.803838 7852 ts_tablet_manager.cc:1428] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260926 23:23:58.804738 7849 ts_tablet_manager.cc:1428] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Time spent starting tablet: real 0.026s user 0.009s sys 0.007s
I20260926 23:23:58.806783 7813 heartbeater.cc:503] Master 127.0.97.62:33391 was elected leader, sending a full tablet report...
W20260926 23:23:58.862084 7815 tablet.cc:2369] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:23:58.939595 7674 tablet.cc:2369] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:58.998370 7857 raft_consensus.cc:482] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:23:58.998693 7857 raft_consensus.cc:504] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:59.000507 7857 leader_election.cc:288] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405), 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491)
I20260926 23:23:59.011674 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.001040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51637 (local address 127.0.97.2:37491)
0926 23:23:59.001261 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:59.001268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:59.002140 (+ 872us) server_negotiation.cc:407] Connection header received
0926 23:23:59.002335 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:59.002341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:59.002424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:59.002539 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:59.003376 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.004578 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.008792 (+ 4214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.010232 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.010407 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.010522 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:59.011218 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 23:23:59.011371 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:23:59.011641 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.001156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50589 (local address 127.0.97.3:41405)
0926 23:23:59.001349 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:59.001358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:59.002140 (+ 782us) server_negotiation.cc:407] Connection header received
0926 23:23:59.002335 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:59.002341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:59.002424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:59.002546 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:23:59.003376 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.004585 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.008792 (+ 4207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.010217 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.010407 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.010522 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:59.011218 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 23:23:59.011371 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:23:59.011936 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.000917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37491 (local address 127.0.97.1:51637)
0926 23:23:59.001948 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:59.001972 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:59.002169 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:59.002658 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:59.002665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:59.002689 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:59.003116 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:59.003133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.004780 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.004783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:59.008471 (+ 3688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:59.008498 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.010578 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.010583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:59.010853 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.011043 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 23:23:59.011166 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 23:23:59.011385 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":399,"spinlock_wait_cycles":10368,"thread_start_us":584,"threads_started":1}
I20260926 23:23:59.012319 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.000908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41405 (local address 127.0.97.1:50589)
0926 23:23:59.001956 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0926 23:23:59.001972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:23:59.002169 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:23:59.002588 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:23:59.002593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:23:59.002638 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:23:59.003197 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:59.003209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.004772 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.004778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:59.008471 (+ 3693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:23:59.008498 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.010568 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.010576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:23:59.010859 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.011043 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 23:23:59.011162 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 23:23:59.011408 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":585,"spinlock_wait_cycles":7424,"thread_start_us":518,"threads_started":1}
I20260926 23:23:59.013126 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3b024717f4d03a482100c4ba6e944" candidate_uuid: "070ebf88cc1b44e49719cc523a51e250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a492260d0b4fb881a25623fde1b624" is_pre_election: true
I20260926 23:23:59.013502 7767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3b024717f4d03a482100c4ba6e944" candidate_uuid: "070ebf88cc1b44e49719cc523a51e250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fccafc1129343ae9e66789b30ee0f66" is_pre_election: true
I20260926 23:23:59.013669 7626 raft_consensus.cc:2454] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 070ebf88cc1b44e49719cc523a51e250 in term 0.
I20260926 23:23:59.014035 7767 raft_consensus.cc:2454] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 070ebf88cc1b44e49719cc523a51e250 in term 0.
I20260926 23:23:59.014436 7418 leader_election.cc:302] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070ebf88cc1b44e49719cc523a51e250, 68a492260d0b4fb881a25623fde1b624; no voters:
I20260926 23:23:59.014926 7857 raft_consensus.cc:2781] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:23:59.015077 7857 raft_consensus.cc:482] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:23:59.015189 7857 raft_consensus.cc:3037] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:59.017556 7857 raft_consensus.cc:504] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:59.018219 7857 leader_election.cc:288] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [CANDIDATE]: Term 1 election: Requested vote from peers 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405), 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491)
I20260926 23:23:59.018811 7767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3b024717f4d03a482100c4ba6e944" candidate_uuid: "070ebf88cc1b44e49719cc523a51e250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fccafc1129343ae9e66789b30ee0f66"
I20260926 23:23:59.018864 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3b024717f4d03a482100c4ba6e944" candidate_uuid: "070ebf88cc1b44e49719cc523a51e250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a492260d0b4fb881a25623fde1b624"
I20260926 23:23:59.019050 7767 raft_consensus.cc:3037] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:59.019419 7626 raft_consensus.cc:3037] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:23:59.022163 7626 raft_consensus.cc:2454] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070ebf88cc1b44e49719cc523a51e250 in term 1.
I20260926 23:23:59.022276 7767 raft_consensus.cc:2454] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070ebf88cc1b44e49719cc523a51e250 in term 1.
I20260926 23:23:59.022682 7418 leader_election.cc:302] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070ebf88cc1b44e49719cc523a51e250, 68a492260d0b4fb881a25623fde1b624; no voters:
I20260926 23:23:59.023039 7857 raft_consensus.cc:2781] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:23:59.023614 7857 raft_consensus.cc:686] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 1 LEADER]: Becoming Leader. State: Replica: 070ebf88cc1b44e49719cc523a51e250, State: Running, Role: LEADER
I20260926 23:23:59.024233 7857 consensus_queue.cc:237] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:23:59.030527 6551 catalog_manager.cc:5495] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 reported cstate change: term changed from 0 to 1, leader changed from <none> to 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "070ebf88cc1b44e49719cc523a51e250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } health_report { overall_health: HEALTHY } } }
W20260926 23:23:59.053949 7533 tablet.cc:2369] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:23:59.118754 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.112829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.0.97.1:39011)
0926 23:23:59.113077 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:59.113084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:59.113221 (+ 137us) server_negotiation.cc:407] Connection header received
0926 23:23:59.113431 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:59.113435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:59.113523 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:59.113666 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:23:59.114075 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.114942 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.116458 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.116982 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.117090 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.117589 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:23:59.118159 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:23:59.118284 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:59.118402 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 23:23:59.118477 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:23:59.127583 7767 raft_consensus.cc:1261] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 FOLLOWER]: Refusing update from remote peer 070ebf88cc1b44e49719cc523a51e250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:59.127583 7626 raft_consensus.cc:1261] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Refusing update from remote peer 070ebf88cc1b44e49719cc523a51e250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:23:59.128598 7857 consensus_queue.cc:1035] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:59.128950 7860 consensus_queue.cc:1035] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:23:59.166918 6551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:59.167289 6551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:23:59.174903 6551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62fec46c64854df8b05a61a9b3831283: Sending DeleteTablet for 3 replicas of tablet 75f3b024717f4d03a482100c4ba6e944
I20260926 23:23:59.176842 7747 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f3b024717f4d03a482100c4ba6e944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44920
I20260926 23:23:59.177870 7873 tablet_replica.cc:330] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: stopping tablet replica
I20260926 23:23:59.178712 7873 raft_consensus.cc:2229] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:59.178830 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f3b024717f4d03a482100c4ba6e944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55834
I20260926 23:23:59.179579 7873 raft_consensus.cc:2258] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:59.180097 7606 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f3b024717f4d03a482100c4ba6e944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:23:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43342
I20260926 23:23:59.181071 7876 tablet_replica.cc:330] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: stopping tablet replica
I20260926 23:23:59.181511 7876 raft_consensus.cc:2229] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:23:59.182076 7876 raft_consensus.cc:2258] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:59.183384 7873 ts_tablet_manager.cc:1905] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:59.183877 7876 ts_tablet_manager.cc:1905] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:59.190856 7876 ts_tablet_manager.cc:1918] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:59.191072 7876 log.cc:1196] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/75f3b024717f4d03a482100c4ba6e944
I20260926 23:23:59.191664 7876 ts_tablet_manager.cc:1939] T 75f3b024717f4d03a482100c4ba6e944 P 68a492260d0b4fb881a25623fde1b624: Deleting consensus metadata
I20260926 23:23:59.193603 6535 catalog_manager.cc:4841] TS 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491): tablet 75f3b024717f4d03a482100c4ba6e944 (table test-table [id=de17cc34e89748c696b769d19da1c8b1]) successfully deleted
I20260926 23:23:59.194445 7873 ts_tablet_manager.cc:1918] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:59.194676 7873 log.cc:1196] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/75f3b024717f4d03a482100c4ba6e944
I20260926 23:23:59.195302 7873 ts_tablet_manager.cc:1939] T 75f3b024717f4d03a482100c4ba6e944 P 2fccafc1129343ae9e66789b30ee0f66: Deleting consensus metadata
I20260926 23:23:59.197615 6537 catalog_manager.cc:4841] TS 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405): tablet 75f3b024717f4d03a482100c4ba6e944 (table test-table [id=de17cc34e89748c696b769d19da1c8b1]) successfully deleted
I20260926 23:23:59.201457 7875 tablet_replica.cc:330] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: stopping tablet replica
I20260926 23:23:59.202219 7875 raft_consensus.cc:2229] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:23:59.203095 7875 raft_consensus.cc:2258] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:23:59.204850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 6519
I20260926 23:23:59.206054 6581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:23:59.208456 7875 ts_tablet_manager.cc:1905] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:23:59.221846 7875 ts_tablet_manager.cc:1918] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:23:59.222312 7875 log.cc:1196] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/75f3b024717f4d03a482100c4ba6e944
I20260926 23:23:59.223145 7875 ts_tablet_manager.cc:1939] T 75f3b024717f4d03a482100c4ba6e944 P 070ebf88cc1b44e49719cc523a51e250: Deleting consensus metadata
I20260926 23:23:59.225018 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.178495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.0.97.61:40969)
0926 23:23:59.178671 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:59.178677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:59.202676 (+ 23999us) server_negotiation.cc:407] Connection header received
0926 23:23:59.202972 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:59.202978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:59.203063 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:59.203141 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:59.206273 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.207382 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.210282 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.210959 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.211112 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.216001 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:59.216009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:59.216014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:59.216083 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:59.220556 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.222101 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.222109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.222113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.222362 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.222929 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.222935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.222939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.223382 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:59.223661 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:59.224326 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 23:23:59.224606 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:23:59.244843 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.202940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.0.97.60:35185)
0926 23:23:59.204838 (+ 1898us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:59.204843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:59.226258 (+ 21415us) server_negotiation.cc:407] Connection header received
0926 23:23:59.226495 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:59.226501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:59.226608 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:59.226739 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:59.228490 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.229784 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.232132 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.232813 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.232959 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.236524 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:59.236534 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:59.236541 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:59.236603 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:59.241296 (+ 4693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.242258 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.242266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.242272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.242500 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.243003 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.243007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.243010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.243432 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:59.243683 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:59.244120 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 23:23:59.244374 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":1628,"threads_started":1}
I20260926 23:23:59.351547 6538 catalog_manager.cc:4841] TS 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011): tablet 75f3b024717f4d03a482100c4ba6e944 (table test-table [id=de17cc34e89748c696b769d19da1c8b1]) successfully deleted
I20260926 23:23:59.359329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 6519
I20260926 23:23:59.405048 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.386384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.0.97.61:40969)
0926 23:23:59.386569 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:59.386578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:59.387520 (+ 942us) server_negotiation.cc:407] Connection header received
0926 23:23:59.387801 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:59.387807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:59.387885 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:59.387960 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:59.390392 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.391519 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.394490 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.395094 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.395256 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.398604 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:59.398609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:59.398613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:59.398647 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:59.401988 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.402844 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.402850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.402854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.402999 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.403451 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.403456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.403460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.403826 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:59.404065 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:59.404429 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 23:23:59.404656 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:23:59.409756 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:23:59.386341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.0.97.60:35185)
0926 23:23:59.386577 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 23:23:59.386586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:23:59.393281 (+ 6695us) server_negotiation.cc:407] Connection header received
0926 23:23:59.394543 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:23:59.394551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:23:59.394638 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:23:59.394757 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:23:59.397169 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.398332 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.400258 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:23:59.400953 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:23:59.401104 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:23:59.403926 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:23:59.403933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:23:59.403937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:23:59.403979 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:23:59.407091 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.407754 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.407759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.407763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.407894 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:23:59.408264 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:23:59.408270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:23:59.408274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:23:59.408554 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:23:59.408838 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 23:23:59.409202 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 23:23:59.409420 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:24:00.151736 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.150955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:60051)
0926 23:24:00.151440 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.151565 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
W20260926 23:24:00.152261 7813 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.158448 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.157728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.1:45373)
0926 23:24:00.158153 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.158287 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
W20260926 23:24:00.158846 7531 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.160986 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.159771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.2:41935)
0926 23:24:00.160607 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.160784 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":614,"threads_started":1}
W20260926 23:24:00.161495 7672 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.676515 7889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62fec46c64854df8b05a61a9b3831283)
I20260926 23:24:00.676887 7889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } }
I20260926 23:24:00.677028 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 62fec46c64854df8b05a61a9b3831283)
I20260926 23:24:00.677286 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } }
I20260926 23:24:00.678401 7889 leader_election.cc:288] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62fec46c64854df8b05a61a9b3831283 (127.0.97.62:33391), 1b16402ace3444f2ba0b2a37688bcf9d (127.0.97.61:40969)
I20260926 23:24:00.678526 7890 leader_election.cc:288] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62fec46c64854df8b05a61a9b3831283 (127.0.97.62:33391), 4d82dd048cc9427c91bb41ed6305ba2a (127.0.97.60:35185)
I20260926 23:24:00.680086 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.678596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34052)
0926 23:24:00.679518 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.679715 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":665,"threads_started":1}
W20260926 23:24:00.681185 7143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.692309 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.679514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.97.61:40969)
0926 23:24:00.681206 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.681213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.681709 (+ 496us) server_negotiation.cc:407] Connection header received
0926 23:24:00.682018 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.682026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.682111 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.682193 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:00.683262 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.685021 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.688732 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.690620 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.690763 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.690917 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.691616 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.691808 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":1572,"thread_start_us":1419,"threads_started":1}
I20260926 23:24:00.693024 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.678614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40969 (local address 127.0.0.1:57248)
0926 23:24:00.681596 (+ 2982us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.681620 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:00.681832 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:00.682593 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:00.682600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:00.682635 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:00.683005 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.683020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.685256 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.685269 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.688468 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.688488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.691080 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.691089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.691384 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.691436 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 23:24:00.692494 (+ 1058us) client_negotiation.cc:240] Negotiation successful
0926 23:24:00.692655 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2868,"threads_started":1}
I20260926 23:24:00.693377 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.678852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34044)
0926 23:24:00.692910 (+ 14058us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.693082 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13920,"thread_start_us":2520,"threads_started":1}
W20260926 23:24:00.694147 7336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.694729 7168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" is_pre_election: true
I20260926 23:24:00.695127 7168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d82dd048cc9427c91bb41ed6305ba2a in term 1.
I20260926 23:24:00.695508 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.678982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35185 (local address 127.0.0.1:55838)
0926 23:24:00.680456 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.680480 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:00.680650 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:00.684480 (+ 3830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:00.684488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:00.684546 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:00.684916 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.684931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.687440 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.687449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.690230 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.690250 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.694583 (+ 4333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.694590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.694824 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.694872 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 23:24:00.694951 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 23:24:00.695098 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"mutex_wait_us":7,"thread_start_us":8156,"threads_started":1}
I20260926 23:24:00.696075 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.695746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34074)
0926 23:24:00.695869 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.695946 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 23:24:00.696089 7335 leader_election.cc:302] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b16402ace3444f2ba0b2a37688bcf9d, 4d82dd048cc9427c91bb41ed6305ba2a; no voters:
I20260926 23:24:00.696502 7889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260926 23:24:00.696540 7143 leader_election.cc:334] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 62fec46c64854df8b05a61a9b3831283 (127.0.97.62:33391): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.696671 7889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 62fec46c64854df8b05a61a9b3831283)
I20260926 23:24:00.696849 7889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:24:00.697003 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.681672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.0.97.60:35185)
0926 23:24:00.683490 (+ 1818us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.683498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.683523 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:24:00.683595 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.683601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.683697 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.685111 (+ 1414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:00.685479 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.687014 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.690493 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.694360 (+ 3867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.695699 (+ 1339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.696259 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.696314 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.696462 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1661,"thread_start_us":1462,"threads_started":1}
I20260926 23:24:00.697340 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.696927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34082)
0926 23:24:00.697110 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.697196 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 23:24:00.697490 7336 leader_election.cc:334] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 62fec46c64854df8b05a61a9b3831283 (127.0.97.62:33391): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.697695 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" is_pre_election: true
I20260926 23:24:00.699324 7889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } }
I20260926 23:24:00.699754 7361 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b16402ace3444f2ba0b2a37688bcf9d in current term 2: Already voted for candidate 4d82dd048cc9427c91bb41ed6305ba2a in this term.
I20260926 23:24:00.699968 7889 leader_election.cc:288] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [CANDIDATE]: Term 2 election: Requested vote from peers 62fec46c64854df8b05a61a9b3831283 (127.0.97.62:33391), 1b16402ace3444f2ba0b2a37688bcf9d (127.0.97.61:40969)
I20260926 23:24:00.700510 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.700055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34092)
0926 23:24:00.700251 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.700353 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 23:24:00.700469 7168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1b16402ace3444f2ba0b2a37688bcf9d"
I20260926 23:24:00.700661 7168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:24:00.700757 7144 leader_election.cc:302] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b16402ace3444f2ba0b2a37688bcf9d; no voters: 4d82dd048cc9427c91bb41ed6305ba2a, 62fec46c64854df8b05a61a9b3831283
I20260926 23:24:00.701485 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.701184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34102)
0926 23:24:00.701323 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.701395 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:24:00.701679 7336 leader_election.cc:334] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 62fec46c64854df8b05a61a9b3831283 (127.0.97.62:33391): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
I20260926 23:24:00.702979 7168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d82dd048cc9427c91bb41ed6305ba2a in term 2.
I20260926 23:24:00.703217 7890 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 23:24:00.703408 7335 leader_election.cc:302] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b16402ace3444f2ba0b2a37688bcf9d, 4d82dd048cc9427c91bb41ed6305ba2a; no voters: 62fec46c64854df8b05a61a9b3831283
I20260926 23:24:00.703689 7889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:24:00.704118 7889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [term 2 LEADER]: Becoming Leader. State: Replica: 4d82dd048cc9427c91bb41ed6305ba2a, State: Running, Role: LEADER
I20260926 23:24:00.704522 7889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } }
I20260926 23:24:00.706403 7899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d82dd048cc9427c91bb41ed6305ba2a. Latest consensus state: current_term: 2 leader_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:24:00.706717 7899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:00.707326 7901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:24:00.710834 7901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:24:00.711651 7901 catalog_manager.cc:1259] Loaded cluster ID: 35e0431d34e74cdfa6af1b055132117a
I20260926 23:24:00.711750 7901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:24:00.712947 7901 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:24:00.713722 7901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Loaded TSK: 0
I20260926 23:24:00.714649 7901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:24:00.753232 7350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:24:00.760646 7350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:24:00.765144 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.764725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34110)
0926 23:24:00.764899 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.765006 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:24:00.765379 7168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [term 2 FOLLOWER]: Refusing update from remote peer 4d82dd048cc9427c91bb41ed6305ba2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 23:24:00.766191 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.765791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34124)
0926 23:24:00.765967 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.766032 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:24:00.766217 7899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 23:24:00.766548 7336 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a -> Peer 62fec46c64854df8b05a61a9b3831283 (127.0.97.62:33391): Couldn't send request to peer 62fec46c64854df8b05a61a9b3831283. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 23:24:00.768857 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d82dd048cc9427c91bb41ed6305ba2a. Latest consensus state: current_term: 2 leader_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:24:00.769171 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:24:00.772799 7889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:24:00.773186 7889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:00.773850 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "62fec46c64854df8b05a61a9b3831283" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } peers { permanent_uuid: "1b16402ace3444f2ba0b2a37688bcf9d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40969 } attrs { promote: false } } peers { permanent_uuid: "4d82dd048cc9427c91bb41ed6305ba2a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35185 } attrs { promote: false } } }
I20260926 23:24:00.774119 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b16402ace3444f2ba0b2a37688bcf9d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:24:00.794147 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.781972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41405 (local address 127.0.0.1:44930)
0926 23:24:00.782143 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.782169 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:00.782318 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:00.782902 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:00.782907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:00.782931 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:00.783282 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.783296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.785496 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.785505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.788399 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.788419 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.792977 (+ 4558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.792987 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.793253 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.793312 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 23:24:00.793396 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 23:24:00.793594 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":1}
I20260926 23:24:00.795655 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.782632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37491 (local address 127.0.0.1:43354)
0926 23:24:00.782851 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.782873 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:00.782990 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:00.784063 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:00.784069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:00.784105 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:00.784307 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.784319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.786457 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.786467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.790604 (+ 4137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.790622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.794423 (+ 3801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.794434 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.794808 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.794879 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 23:24:00.794970 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 23:24:00.795173 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":1453}
I20260926 23:24:00.796597 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.783174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.0.97.2:37491)
0926 23:24:00.783681 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.783689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.783716 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:24:00.783799 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.783804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.783893 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.784461 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:00.784629 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.786236 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.790828 (+ 4592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.794173 (+ 3345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.795970 (+ 1797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.796125 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.796196 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.796336 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":262,"threads_started":1}
I20260926 23:24:00.796588 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.782066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.0.97.3:41405)
0926 23:24:00.782550 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.782556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.782573 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:24:00.782642 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.782648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.782731 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.782880 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:00.783816 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.785295 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.790920 (+ 5625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.792630 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.795364 (+ 2734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.795515 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.795580 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.796175 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
I20260926 23:24:00.797533 7606 tablet_service.cc:1467] Processing CreateTablet for tablet a171d2b4eb9c4240b897d9cc21c50a57 (DEFAULT_TABLE table=test-table [id=d4fb273810d7457caa05ff155d7bdb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:00.797502 7747 tablet_service.cc:1467] Processing CreateTablet for tablet a171d2b4eb9c4240b897d9cc21c50a57 (DEFAULT_TABLE table=test-table [id=d4fb273810d7457caa05ff155d7bdb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:00.798018 7606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a171d2b4eb9c4240b897d9cc21c50a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:00.798066 7747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a171d2b4eb9c4240b897d9cc21c50a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:00.808888 7913 tablet_bootstrap.cc:492] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: Bootstrap starting.
I20260926 23:24:00.811452 7913 tablet_bootstrap.cc:654] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:00.814222 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.782835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39011 (local address 127.0.0.1:55854)
0926 23:24:00.794461 (+ 11626us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.794484 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:00.794635 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:00.807183 (+ 12548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:00.807188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:00.807224 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:00.807446 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.807464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.809131 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.809136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.811050 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.811065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.813103 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.813113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.813408 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.813467 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 23:24:00.813550 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 23:24:00.813739 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11492,"thread_start_us":13345,"threads_started":1}
I20260926 23:24:00.815318 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.783430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.0.97.1:39011)
0926 23:24:00.806861 (+ 23431us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.806866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.806884 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:24:00.806961 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.806965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.807032 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.807160 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:00.807664 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.808980 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.811353 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.812891 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.814461 (+ 1570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.814594 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.814659 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.814847 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23314,"thread_start_us":23194,"threads_started":1}
I20260926 23:24:00.816395 7465 tablet_service.cc:1467] Processing CreateTablet for tablet a171d2b4eb9c4240b897d9cc21c50a57 (DEFAULT_TABLE table=test-table [id=d4fb273810d7457caa05ff155d7bdb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:00.817106 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a171d2b4eb9c4240b897d9cc21c50a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:00.828326 7912 tablet_bootstrap.cc:492] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: Bootstrap starting.
I20260926 23:24:00.830869 7912 tablet_bootstrap.cc:654] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:00.831620 7915 tablet_bootstrap.cc:492] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: Bootstrap starting.
I20260926 23:24:00.833428 7913 tablet_bootstrap.cc:492] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: No bootstrap required, opened a new log
I20260926 23:24:00.833642 7913 ts_tablet_manager.cc:1397] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.008s
I20260926 23:24:00.833827 7915 tablet_bootstrap.cc:654] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:00.833972 7912 tablet_bootstrap.cc:492] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: No bootstrap required, opened a new log
I20260926 23:24:00.834177 7912 ts_tablet_manager.cc:1397] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 23:24:00.834591 7913 raft_consensus.cc:348] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.834920 7913 raft_consensus.cc:374] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:00.835045 7913 raft_consensus.cc:729] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fccafc1129343ae9e66789b30ee0f66, State: Initialized, Role: FOLLOWER
I20260926 23:24:00.835173 7912 raft_consensus.cc:348] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.835356 7913 consensus_queue.cc:260] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.835570 7912 raft_consensus.cc:374] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:00.835714 7912 raft_consensus.cc:729] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a492260d0b4fb881a25623fde1b624, State: Initialized, Role: FOLLOWER
I20260926 23:24:00.836091 7912 consensus_queue.cc:260] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.839116 7912 ts_tablet_manager.cc:1428] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 23:24:00.842298 7915 tablet_bootstrap.cc:492] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: No bootstrap required, opened a new log
I20260926 23:24:00.842514 7915 ts_tablet_manager.cc:1397] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260926 23:24:00.843602 7671 heartbeater.cc:503] Master 127.0.97.60:35185 was elected leader, sending a full tablet report...
I20260926 23:24:00.846313 7913 ts_tablet_manager.cc:1428] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: Time spent starting tablet: real 0.012s user 0.001s sys 0.003s
I20260926 23:24:00.846621 7812 heartbeater.cc:503] Master 127.0.97.60:35185 was elected leader, sending a full tablet report...
I20260926 23:24:00.850224 7915 raft_consensus.cc:348] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.850458 7915 raft_consensus.cc:374] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:00.850497 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.844346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:44189)
0926 23:24:00.850019 (+ 5673us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.850179 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5488,"thread_start_us":5317,"threads_started":1}
I20260926 23:24:00.850574 7915 raft_consensus.cc:729] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070ebf88cc1b44e49719cc523a51e250, State: Initialized, Role: FOLLOWER
I20260926 23:24:00.850942 7915 consensus_queue.cc:260] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.851552 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.851164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:35787)
0926 23:24:00.851344 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.851419 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 23:24:00.851837 7915 ts_tablet_manager.cc:1428] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260926 23:24:00.853807 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.839832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.2:49421)
0926 23:24:00.853347 (+ 13515us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.853505 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13362,"thread_start_us":13206,"threads_started":1}
I20260926 23:24:00.854542 7530 heartbeater.cc:503] Master 127.0.97.60:35185 was elected leader, sending a full tablet report...
I20260926 23:24:00.855017 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.852491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.1:53225)
0926 23:24:00.854517 (+ 2026us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.854701 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1899,"thread_start_us":1749,"threads_started":1}
I20260926 23:24:00.855484 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.855104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.2:40381)
0926 23:24:00.855249 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.855345 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 23:24:00.856184 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.855830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.1:40825)
0926 23:24:00.856006 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.856077 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 23:24:00.868887 7918 raft_consensus.cc:482] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:00.869174 7918 raft_consensus.cc:504] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.870795 7918 leader_election.cc:288] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491), 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011)
I20260926 23:24:00.878854 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.871193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50253 (local address 127.0.97.1:39011)
0926 23:24:00.871376 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.871382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.871909 (+ 527us) server_negotiation.cc:407] Connection header received
0926 23:24:00.872047 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.872051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.872108 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.872177 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:00.872730 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.873757 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.876170 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.877672 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.877816 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.877935 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.878429 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.878573 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:24:00.878854 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.871193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43015 (local address 127.0.97.2:37491)
0926 23:24:00.871456 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.871466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.871493 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:24:00.871656 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.871662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.871752 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.871831 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:00.872367 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.873419 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.876170 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.877643 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.877826 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.877935 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.878429 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.878573 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:24:00.878952 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.871024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37491 (local address 127.0.97.3:43015)
0926 23:24:00.871281 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.871301 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:00.871494 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:00.871884 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:00.871889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:00.871936 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:00.872216 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.872225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.873564 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.873568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.875960 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.875975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.877906 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.877913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.878168 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.878254 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 23:24:00.878382 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 23:24:00.878536 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":164,"spinlock_wait_cycles":7168}
I20260926 23:24:00.878996 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.871059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39011 (local address 127.0.97.3:50253)
0926 23:24:00.871770 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.871783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:00.871938 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:00.872225 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:00.872230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:00.872255 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:00.872553 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.872562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.873870 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.873876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.875959 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:00.875976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.877906 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.877913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:00.878163 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.878260 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 23:24:00.878358 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 23:24:00.878536 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":85,"spinlock_wait_cycles":4224,"thread_start_us":458,"threads_started":1}
I20260926 23:24:00.879662 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171d2b4eb9c4240b897d9cc21c50a57" candidate_uuid: "2fccafc1129343ae9e66789b30ee0f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a492260d0b4fb881a25623fde1b624" is_pre_election: true
I20260926 23:24:00.879926 7626 raft_consensus.cc:2454] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fccafc1129343ae9e66789b30ee0f66 in term 0.
I20260926 23:24:00.879953 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171d2b4eb9c4240b897d9cc21c50a57" candidate_uuid: "2fccafc1129343ae9e66789b30ee0f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070ebf88cc1b44e49719cc523a51e250" is_pre_election: true
I20260926 23:24:00.880414 7485 raft_consensus.cc:2454] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fccafc1129343ae9e66789b30ee0f66 in term 0.
I20260926 23:24:00.880494 7700 leader_election.cc:302] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fccafc1129343ae9e66789b30ee0f66, 68a492260d0b4fb881a25623fde1b624; no voters:
I20260926 23:24:00.880944 7918 raft_consensus.cc:2781] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:24:00.881075 7918 raft_consensus.cc:482] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:24:00.881177 7918 raft_consensus.cc:3037] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:00.883548 7918 raft_consensus.cc:504] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.884078 7918 leader_election.cc:288] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [CANDIDATE]: Term 1 election: Requested vote from peers 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491), 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011)
I20260926 23:24:00.884522 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171d2b4eb9c4240b897d9cc21c50a57" candidate_uuid: "2fccafc1129343ae9e66789b30ee0f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070ebf88cc1b44e49719cc523a51e250"
I20260926 23:24:00.884555 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a171d2b4eb9c4240b897d9cc21c50a57" candidate_uuid: "2fccafc1129343ae9e66789b30ee0f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a492260d0b4fb881a25623fde1b624"
I20260926 23:24:00.884723 7485 raft_consensus.cc:3037] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:00.884845 7626 raft_consensus.cc:3037] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:00.886793 7485 raft_consensus.cc:2454] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fccafc1129343ae9e66789b30ee0f66 in term 1.
I20260926 23:24:00.887032 7626 raft_consensus.cc:2454] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fccafc1129343ae9e66789b30ee0f66 in term 1.
I20260926 23:24:00.887180 7703 leader_election.cc:302] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070ebf88cc1b44e49719cc523a51e250, 2fccafc1129343ae9e66789b30ee0f66; no voters:
I20260926 23:24:00.887517 7918 raft_consensus.cc:2781] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:00.888074 7918 raft_consensus.cc:686] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 LEADER]: Becoming Leader. State: Replica: 2fccafc1129343ae9e66789b30ee0f66, State: Running, Role: LEADER
I20260926 23:24:00.888594 7918 consensus_queue.cc:237] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } }
I20260926 23:24:00.889389 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.888943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:47989)
0926 23:24:00.889162 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.889260 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 23:24:00.892325 7350 catalog_manager.cc:5495] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2fccafc1129343ae9e66789b30ee0f66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fccafc1129343ae9e66789b30ee0f66" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 } health_report { overall_health: UNKNOWN } } }
I20260926 23:24:00.924065 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.919149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.0.97.3:41405)
0926 23:24:00.919338 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.919343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.919360 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:24:00.919486 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.919489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.919578 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.919699 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:24:00.920041 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.920817 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.922196 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.922589 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.922678 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.923040 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:24:00.923593 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:24:00.923697 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:00.923811 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 23:24:00.923873 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:24:00.928258 7626 raft_consensus.cc:1261] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Refusing update from remote peer 2fccafc1129343ae9e66789b30ee0f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:00.928524 7485 raft_consensus.cc:1261] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 1 FOLLOWER]: Refusing update from remote peer 2fccafc1129343ae9e66789b30ee0f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:00.929147 7925 consensus_queue.cc:1035] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a492260d0b4fb881a25623fde1b624" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:00.929595 7918 consensus_queue.cc:1035] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070ebf88cc1b44e49719cc523a51e250" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:00.931852 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.931272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.2:54299)
0926 23:24:00.931496 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.931620 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 23:24:00.932631 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.932062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.1:33941)
0926 23:24:00.932328 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.932453 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260926 23:24:00.942716 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.942206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:41185)
0926 23:24:00.942425 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.942520 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 23:24:00.944542 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.944004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.1:50357)
0926 23:24:00.944276 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.944385 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260926 23:24:00.952358 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.951880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.3:56211)
0926 23:24:00.952091 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.952195 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 23:24:00.955230 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.954600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.97.2:43091)
0926 23:24:00.954801 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:00.955046 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 23:24:00.961777 7351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:00.962091 7351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:00.968767 7351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d82dd048cc9427c91bb41ed6305ba2a: Sending DeleteTablet for 3 replicas of tablet a171d2b4eb9c4240b897d9cc21c50a57
I20260926 23:24:00.969810 7747 tablet_service.cc:1514] Processing DeleteTablet for tablet a171d2b4eb9c4240b897d9cc21c50a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:00 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44930
I20260926 23:24:00.970173 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet a171d2b4eb9c4240b897d9cc21c50a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:00 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55854
I20260926 23:24:00.970280 7606 tablet_service.cc:1514] Processing DeleteTablet for tablet a171d2b4eb9c4240b897d9cc21c50a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:00 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:43354
I20260926 23:24:00.970607 7937 tablet_replica.cc:330] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: stopping tablet replica
I20260926 23:24:00.970907 7937 raft_consensus.cc:2229] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:24:00.971825 7937 raft_consensus.cc:2258] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:00.973512 7937 ts_tablet_manager.cc:1905] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:00.975274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 7326
I20260926 23:24:00.981258 7938 tablet_replica.cc:330] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: stopping tablet replica
I20260926 23:24:00.981520 7938 raft_consensus.cc:2229] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:00.981894 7938 raft_consensus.cc:2258] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:00.982424 7939 tablet_replica.cc:330] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: stopping tablet replica
I20260926 23:24:00.982640 7939 raft_consensus.cc:2229] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:00.982931 7939 raft_consensus.cc:2258] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:00.984647 7939 ts_tablet_manager.cc:1905] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:00.986511 7938 ts_tablet_manager.cc:1905] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:00.988343 7937 ts_tablet_manager.cc:1918] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:00.988515 7937 log.cc:1196] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/a171d2b4eb9c4240b897d9cc21c50a57
I20260926 23:24:00.989051 7937 ts_tablet_manager.cc:1939] T a171d2b4eb9c4240b897d9cc21c50a57 P 2fccafc1129343ae9e66789b30ee0f66: Deleting consensus metadata
I20260926 23:24:00.990149 7337 catalog_manager.cc:4841] TS 2fccafc1129343ae9e66789b30ee0f66 (127.0.97.3:41405): tablet a171d2b4eb9c4240b897d9cc21c50a57 (table test-table [id=d4fb273810d7457caa05ff155d7bdb5c]) successfully deleted
I20260926 23:24:00.993273 7939 ts_tablet_manager.cc:1918] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:00.993417 7939 log.cc:1196] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/a171d2b4eb9c4240b897d9cc21c50a57
I20260926 23:24:00.993990 7939 ts_tablet_manager.cc:1939] T a171d2b4eb9c4240b897d9cc21c50a57 P 68a492260d0b4fb881a25623fde1b624: Deleting consensus metadata
I20260926 23:24:00.994347 7938 ts_tablet_manager.cc:1918] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:00.994544 7938 log.cc:1196] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/a171d2b4eb9c4240b897d9cc21c50a57
I20260926 23:24:00.995086 7938 ts_tablet_manager.cc:1939] T a171d2b4eb9c4240b897d9cc21c50a57 P 070ebf88cc1b44e49719cc523a51e250: Deleting consensus metadata
I20260926 23:24:00.995349 7335 catalog_manager.cc:4841] TS 68a492260d0b4fb881a25623fde1b624 (127.0.97.2:37491): tablet a171d2b4eb9c4240b897d9cc21c50a57 (table test-table [id=d4fb273810d7457caa05ff155d7bdb5c]) successfully deleted
I20260926 23:24:00.996446 7338 catalog_manager.cc:4841] TS 070ebf88cc1b44e49719cc523a51e250 (127.0.97.1:39011): tablet a171d2b4eb9c4240b897d9cc21c50a57 (table test-table [id=d4fb273810d7457caa05ff155d7bdb5c]) successfully deleted
I20260926 23:24:01.004340 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:00.976738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.0.97.1:39011)
0926 23:24:00.976937 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:00.976944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:00.983496 (+ 6552us) server_negotiation.cc:407] Connection header received
0926 23:24:00.983721 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:00.983728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:00.983818 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:00.983937 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:00.985888 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.987136 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.990671 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:00.991298 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:00.991409 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:00.996964 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:00.996993 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:00.997021 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:00.997068 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:01.001270 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:01.001985 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:01.001996 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:01.002000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:01.002176 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:01.002558 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:01.002564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:01.002568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:01.003077 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:01.003423 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:01.003691 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 23:24:01.003931 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:24:01.005012 7525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:01.207229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 7326
I20260926 23:24:01.229884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 7542
I20260926 23:24:01.245348 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:01.230747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.0.97.2:37491)
0926 23:24:01.230937 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:01.230942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:01.231039 (+ 97us) server_negotiation.cc:407] Connection header received
0926 23:24:01.231229 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:01.231233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:01.231291 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:01.231372 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:01.233085 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:01.233839 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:01.235645 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:01.236015 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:01.236106 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:01.239304 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:01.239317 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:01.239333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:01.239364 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:01.242298 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:01.243047 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:01.243054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:01.243056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:01.243228 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:01.243657 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:01.243662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:01.243664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:01.244082 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:01.244367 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:01.244630 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 23:24:01.244881 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:24:01.246165 7666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:01.294164 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:01.293619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34132)
0926 23:24:01.293849 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:01.293960 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:24:01.297487 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:01.297048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34148)
0926 23:24:01.297234 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:01.297333 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 23:24:01.344667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 7542
I20260926 23:24:01.368587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 7684
I20260926 23:24:01.383528 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:01.369517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.0.97.3:41405)
0926 23:24:01.369760 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:01.369765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:01.369785 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:24:01.369955 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:01.369967 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:01.370032 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:01.370140 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:01.371439 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:01.372257 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:01.373737 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:01.374223 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:01.374366 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:01.376827 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:01.376839 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:01.376855 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:01.376888 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:01.380381 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:01.381199 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:01.381210 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:01.381219 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:01.381386 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:01.381862 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:01.381868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:01.381873 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:01.382263 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:01.382624 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:01.382926 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 23:24:01.383193 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 23:24:01.384387 7807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:01.513595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 7684
I20260926 23:24:01.533933 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 7014
I20260926 23:24:01.535015 7188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:01.648662 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 7014
I20260926 23:24:01.669466 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 7115
I20260926 23:24:01.670562 7381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:01.789500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 7115
I20260926 23:24:01.789637 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:01.789130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33391 (local address 127.0.0.1:34150)
0926 23:24:01.789357 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:01.789465 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
2026-09-26T23:24:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10350 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:24:01 dist-test-slave-q5g8 krb5kdc[7949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:24:01 dist-test-slave-q5g8 krb5kdc[7949](info): set up 2 sockets
Sep 26 23:24:01 dist-test-slave-q5g8 krb5kdc[7949](info): commencing operation
krb5kdc: starting...
W20260926 23:24:02.881572 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:24:03 dist-test-slave-q5g8 krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:24:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:24:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 23:24:03.089308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35845
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:03.402480 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:03.402799 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:03.402962 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:03.409962 7962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:24:03.410120 7962 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:24:03.410214 7962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:03.410336 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:03.410421 7962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:24:03.410511 7962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:24:03.419932 7962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:03.421239 7962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:03.422544 7962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:03.434185 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:03.434269 7970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:03.434293 7968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:03.434933 7962 server_base.cc:1034] running on GCE node
I20260926 23:24:03.435741 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:03.437150 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:03.438359 7962 hybrid_clock.cc:648] HybridClock initialized: now 1790465043438303 us; error 28 us; skew 500 ppm
I20260926 23:24:03.438699 7962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:03 dist-test-slave-q5g8 krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:03.441790 7962 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 23:24:03.443181 7962 webserver.cc:460] Webserver started at http://127.0.97.62:33539/ using document root <none> and password file <none>
I20260926 23:24:03.443862 7962 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:03.443991 7962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:03.444306 7962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:03.447175 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "66518814d1724995add55f1119f9db43"
format_stamp: "Formatted at 2026-09-26 23:24:03 on dist-test-slave-q5g8"
server_key: "4de314f1d986dc5c49f35fc6544e209c"
server_key_iv: "4def753b76caa709f54c2293591e163d"
server_key_version: "encryptionkey@0"
I20260926 23:24:03.447983 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "66518814d1724995add55f1119f9db43"
format_stamp: "Formatted at 2026-09-26 23:24:03 on dist-test-slave-q5g8"
server_key: "4de314f1d986dc5c49f35fc6544e209c"
server_key_iv: "4def753b76caa709f54c2293591e163d"
server_key_version: "encryptionkey@0"
I20260926 23:24:03.453073 7962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 23:24:03.456365 7977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:03.457679 7962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:03.457863 7962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "66518814d1724995add55f1119f9db43"
format_stamp: "Formatted at 2026-09-26 23:24:03 on dist-test-slave-q5g8"
server_key: "4de314f1d986dc5c49f35fc6544e209c"
server_key_iv: "4def753b76caa709f54c2293591e163d"
server_key_version: "encryptionkey@0"
I20260926 23:24:03.458076 7962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:03.471470 7962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:03.475641 7962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:03.475900 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:03.497215 7962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35845
I20260926 23:24:03.497236 8038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35845 every 8 connection(s)
I20260926 23:24:03.498546 7962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:24:03.499073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 7962
I20260926 23:24:03.499315 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:24:03.500168 388 external_mini_cluster.cc:1428] Setting key 67c93edbf3acf67663d975ec7e640ab6
I20260926 23:24:03.502894 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:24:03 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:24:03.515748 8039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Bootstrap starting.
I20260926 23:24:03.518788 8039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:03.519804 8039 log.cc:824] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:03.522598 8039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: No bootstrap required, opened a new log
I20260926 23:24:03.523059 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:03.502417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.0.97.62:35845)
0926 23:24:03.503261 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:03.503274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:03.503317 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:24:03.505028 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:03.505049 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:03.505456 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:03.505794 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:03.507565 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:03.509047 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:03.510726 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:03.511352 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:03.511531 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:03.515997 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:03.516006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:03.516019 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:03.516059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:03.519073 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:03.519741 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:03.519747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:03.519753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:03.519970 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:03.520345 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:03.520350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:03.520354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:03.520915 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:03.521207 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:03.521640 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 23:24:03.522061 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":402,"threads_started":1}
I20260926 23:24:03.530776 8039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66518814d1724995add55f1119f9db43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35845 } }
I20260926 23:24:03.531145 8039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:03.531252 8039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66518814d1724995add55f1119f9db43, State: Initialized, Role: FOLLOWER
I20260926 23:24:03.531780 8039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66518814d1724995add55f1119f9db43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35845 } }
I20260926 23:24:03.532020 8039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:24:03.532171 8039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:24:03.532358 8039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:03.534500 8039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66518814d1724995add55f1119f9db43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35845 } }
I20260926 23:24:03.534986 8039 leader_election.cc:302] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66518814d1724995add55f1119f9db43; no voters:
I20260926 23:24:03.535567 8039 leader_election.cc:288] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:24:03.535705 8044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:03.536402 8044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [term 1 LEADER]: Becoming Leader. State: Replica: 66518814d1724995add55f1119f9db43, State: Running, Role: LEADER
I20260926 23:24:03.536972 8044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66518814d1724995add55f1119f9db43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35845 } }
I20260926 23:24:03.537768 8039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:24:03.540249 8045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66518814d1724995add55f1119f9db43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66518814d1724995add55f1119f9db43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35845 } } }
I20260926 23:24:03.540136 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66518814d1724995add55f1119f9db43. Latest consensus state: current_term: 1 leader_uuid: "66518814d1724995add55f1119f9db43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66518814d1724995add55f1119f9db43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35845 } } }
I20260926 23:24:03.540601 8045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:03.540630 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43 [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:03.548174 8053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:24:03.553210 8053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:24:03.561501 8053 catalog_manager.cc:1347] Generated new cluster ID: f50d633f4ff545abbe24621d540bf32c
I20260926 23:24:03.561621 8053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:24:03.589120 8053 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:24:03.590499 8053 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:24:03.597487 8053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Generated new TSK 0
I20260926 23:24:03.598338 8053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 23:24:03.668843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35845
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:03.995117 8065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:03.995375 8065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:03.995469 8065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:04.002528 8065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:04.002631 8065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:04.002719 8065 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:24:04.002799 8065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:24:04.012415 8065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:04.013676 8065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:04.014952 8065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:04.029001 8070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:04.028904 8073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:04.028904 8071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:04.029840 8065 server_base.cc:1034] running on GCE node
I20260926 23:24:04.030656 8065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:04.031657 8065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:04.032966 8065 hybrid_clock.cc:648] HybridClock initialized: now 1790465044032850 us; error 35 us; skew 500 ppm
I20260926 23:24:04.033277 8065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:04 dist-test-slave-q5g8 krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:04.036350 8065 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:24:04.037899 8065 webserver.cc:460] Webserver started at http://127.0.97.1:36293/ using document root <none> and password file <none>
I20260926 23:24:04.038611 8065 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:04.038709 8065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:04.039078 8065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:04.041935 8065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "f583e7ac98ff4f2eb80d1d783ad3b338"
format_stamp: "Formatted at 2026-09-26 23:24:04 on dist-test-slave-q5g8"
server_key: "bd67ec8ace08f4bb0d79cd46d221a877"
server_key_iv: "8f4437e1a52f136700103fde2abbdee5"
server_key_version: "encryptionkey@0"
I20260926 23:24:04.042726 8065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f583e7ac98ff4f2eb80d1d783ad3b338"
format_stamp: "Formatted at 2026-09-26 23:24:04 on dist-test-slave-q5g8"
server_key: "bd67ec8ace08f4bb0d79cd46d221a877"
server_key_iv: "8f4437e1a52f136700103fde2abbdee5"
server_key_version: "encryptionkey@0"
I20260926 23:24:04.047883 8065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260926 23:24:04.051403 8080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:04.052665 8065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:24:04.052915 8065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "f583e7ac98ff4f2eb80d1d783ad3b338"
format_stamp: "Formatted at 2026-09-26 23:24:04 on dist-test-slave-q5g8"
server_key: "bd67ec8ace08f4bb0d79cd46d221a877"
server_key_iv: "8f4437e1a52f136700103fde2abbdee5"
server_key_version: "encryptionkey@0"
I20260926 23:24:04.053133 8065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:04.066804 8065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:04.070556 8065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:04.070824 8065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:04.073256 8065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:04.073333 8065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:04.073508 8065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:04.073581 8065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:24:04 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:24:04.098970 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.073978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49015 (local address 127.0.97.62:35845)
0926 23:24:04.074472 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:04.074479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:04.075466 (+ 987us) server_negotiation.cc:407] Connection header received
0926 23:24:04.077651 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:04.077659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:04.077768 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:04.077898 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:04.080069 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.081415 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.083959 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.084379 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.084525 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.089958 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:04.089963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:04.089967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:04.090001 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:04.095009 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.095849 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.095857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.095864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.096030 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.096718 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.096727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.096733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.097168 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:04.097412 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:04.098483 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0926 23:24:04.098673 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":261,"threads_started":1}
I20260926 23:24:04.101168 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.074430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35845 (local address 127.0.97.1:49015)
0926 23:24:04.075282 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:04.075335 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:04.077346 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:04.078156 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:04.078167 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:04.079003 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:04.079839 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.079870 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.081600 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.081607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.083773 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.083787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.084621 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.084631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.084954 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.085890 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:04.085924 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:04.089675 (+ 3751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:04.095189 (+ 5514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.095200 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.095220 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.095711 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.096147 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.096152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.096156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.096541 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.097388 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:04.097408 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:04.097987 (+ 579us) client_negotiation.cc:769] Sending connection context
0926 23:24:04.098425 (+ 438us) client_negotiation.cc:240] Negotiation successful
0926 23:24:04.099160 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":361,"threads_started":1}
I20260926 23:24:04.122661 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.106922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40683 (local address 127.0.97.62:35845)
0926 23:24:04.107143 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:04.107152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:04.107798 (+ 646us) server_negotiation.cc:407] Connection header received
0926 23:24:04.108049 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:04.108056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:04.108144 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:04.108256 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:04.109680 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.110638 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.112347 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.113208 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.113363 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.116115 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:04.116126 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:04.116129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:04.116158 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:04.119120 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.120000 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.120004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.120006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.120146 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.120576 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.120580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.120582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.121028 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:04.121212 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:04.121809 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 23:24:04.122152 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:24:04.122815 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.106896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35845 (local address 127.0.97.1:40683)
0926 23:24:04.107673 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:04.107695 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:04.107885 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:04.108299 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:04.108304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:04.109001 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:04.109463 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.109482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.110813 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.110817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.112201 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.112212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.113435 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.113443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.113645 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.114393 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:04.114435 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:04.115952 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:04.119268 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.119275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.119279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.119874 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.120272 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.120280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.120285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.120461 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.121181 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:04.121189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:04.121454 (+ 265us) client_negotiation.cc:769] Sending connection context
0926 23:24:04.121658 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 23:24:04.122238 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":406,"threads_started":1}
I20260926 23:24:04.130254 8065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43483
I20260926 23:24:04.130260 8200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43483 every 8 connection(s)
I20260926 23:24:04.131662 8065 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:24:04.133518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 8065
I20260926 23:24:04.133759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:24:04.134008 8201 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35845
I20260926 23:24:04.134269 8201 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:04.134720 388 external_mini_cluster.cc:1428] Setting key 974dc6a0e422de912753e76cf80b825d
I20260926 23:24:04.135144 8201 heartbeater.cc:511] Master 127.0.97.62:35845 requested a full tablet report, sending...
I20260926 23:24:04.137476 7993 ts_manager.cc:194] Registered new tserver with Master: f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483)
I20260926 23:24:04.139045 7993 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49015
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 23:24:04.183687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35845
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:04.507409 8205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:04.507814 8205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:04.507956 8205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:04.519618 8205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:04.519764 8205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:04.519889 8205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:24:04.520015 8205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:24:04.530258 8205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:04.531494 8205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:04.532997 8205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:04.545849 8210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:04.545871 8213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:04.546120 8211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:04.547418 8205 server_base.cc:1034] running on GCE node
I20260926 23:24:04.548107 8205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:04.549226 8205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:04.550486 8205 hybrid_clock.cc:648] HybridClock initialized: now 1790465044550420 us; error 43 us; skew 500 ppm
I20260926 23:24:04.550880 8205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:04 dist-test-slave-q5g8 krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:04.554266 8205 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:24:04.555729 8205 webserver.cc:460] Webserver started at http://127.0.97.2:39981/ using document root <none> and password file <none>
I20260926 23:24:04.556584 8205 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:04.556740 8205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:04.557129 8205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:04.560163 8205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "c553c6025f1a42e3a050f2e7ea31cde2"
format_stamp: "Formatted at 2026-09-26 23:24:04 on dist-test-slave-q5g8"
server_key: "bdacff716c71440ea89bb238bbc23953"
server_key_iv: "ce5e6a99dbb505c5102a66c0f2a0904a"
server_key_version: "encryptionkey@0"
I20260926 23:24:04.561096 8205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c553c6025f1a42e3a050f2e7ea31cde2"
format_stamp: "Formatted at 2026-09-26 23:24:04 on dist-test-slave-q5g8"
server_key: "bdacff716c71440ea89bb238bbc23953"
server_key_iv: "ce5e6a99dbb505c5102a66c0f2a0904a"
server_key_version: "encryptionkey@0"
I20260926 23:24:04.566280 8205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:24:04.569931 8220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:04.571401 8205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:24:04.571597 8205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "c553c6025f1a42e3a050f2e7ea31cde2"
format_stamp: "Formatted at 2026-09-26 23:24:04 on dist-test-slave-q5g8"
server_key: "bdacff716c71440ea89bb238bbc23953"
server_key_iv: "ce5e6a99dbb505c5102a66c0f2a0904a"
server_key_version: "encryptionkey@0"
I20260926 23:24:04.571837 8205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:04.598907 8205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:04.603526 8205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:04.603821 8205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:04.606827 8205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:04.606941 8205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:04.607102 8205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:04.607178 8205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:24:04 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:24:04.630975 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.607521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44807 (local address 127.0.97.62:35845)
0926 23:24:04.607710 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:04.607716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:04.609167 (+ 1451us) server_negotiation.cc:407] Connection header received
0926 23:24:04.611193 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:04.611199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:04.611293 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:04.611399 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:04.613658 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.614971 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.617534 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.618183 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.618344 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.624139 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:04.624145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:04.624149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:04.624182 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:04.627406 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.628528 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.628536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.628541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.628731 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.629214 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.629220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.629223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.629502 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:04.629668 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:04.630376 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 23:24:04.630595 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:24:04.632052 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.608070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35845 (local address 127.0.97.2:44807)
0926 23:24:04.608962 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:04.609024 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:04.610897 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:04.611655 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:04.611668 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:04.612506 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:04.613423 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.613462 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.615175 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.615182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.617372 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.617383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.618688 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.618694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.619014 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.619824 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:04.619864 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:04.623823 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:04.627565 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.627574 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.627596 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.628346 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.628859 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.628864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.628868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.629092 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.629628 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:04.629638 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:04.630005 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 23:24:04.630265 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 23:24:04.630810 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":399,"threads_started":1}
I20260926 23:24:04.655755 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.639845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50825 (local address 127.0.97.62:35845)
0926 23:24:04.640044 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:04.640052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:04.640734 (+ 682us) server_negotiation.cc:407] Connection header received
0926 23:24:04.640938 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:04.640941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:04.641008 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:04.641085 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:04.642533 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.643228 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.645714 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.646183 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.646338 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.648980 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:04.648986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:04.648992 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:04.649036 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:04.652821 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.653710 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.653714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.653716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.653827 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:04.654276 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:04.654283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:04.654286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:04.654651 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:04.654804 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:04.655186 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 23:24:04.655401 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:24:04.655859 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:04.639830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35845 (local address 127.0.97.2:50825)
0926 23:24:04.640572 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:04.640598 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:04.640804 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:04.641150 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:04.641158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:04.641924 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:04.642382 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.642399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.643365 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.643373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.645558 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:04.645579 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:04.646415 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:04.646425 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:04.646685 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:04.647385 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:04.647418 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:04.648746 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:04.653007 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.653011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.653014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.653573 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.653937 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:04.653941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:04.653945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:04.654127 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:04.654773 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:04.654778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:04.654946 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 23:24:04.655089 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:24:04.655407 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":376,"threads_started":1}
I20260926 23:24:04.667318 8205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38325
I20260926 23:24:04.667326 8339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38325 every 8 connection(s)
I20260926 23:24:04.668586 8205 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:24:04.670655 8340 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35845
I20260926 23:24:04.670846 8340 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:04.671429 8340 heartbeater.cc:511] Master 127.0.97.62:35845 requested a full tablet report, sending...
I20260926 23:24:04.672781 7993 ts_manager.cc:194] Registered new tserver with Master: c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325)
I20260926 23:24:04.673694 7993 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44807
I20260926 23:24:04.675113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 8205
I20260926 23:24:04.675311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:24:04.675757 388 external_mini_cluster.cc:1428] Setting key 9786d55b465b6e2482b1981291e81379
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 23:24:04.724884 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35845
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:05.056654 8344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:05.057096 8344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:05.057199 8344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:05.065430 8344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:05.065564 8344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:05.065732 8344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:24:05.065881 8344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:24:05.076278 8344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:05.077690 8344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:05.079099 8344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:05.092934 8350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:05.092941 8349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:05.093139 8352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:05.093843 8344 server_base.cc:1034] running on GCE node
I20260926 23:24:05.094472 8344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:05.095486 8344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:05.096779 8344 hybrid_clock.cc:648] HybridClock initialized: now 1790465045096704 us; error 43 us; skew 500 ppm
I20260926 23:24:05.097083 8344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:05 dist-test-slave-q5g8 krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465045, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:05.100548 8344 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:24:05.102106 8344 webserver.cc:460] Webserver started at http://127.0.97.3:37881/ using document root <none> and password file <none>
I20260926 23:24:05.102872 8344 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:05.102962 8344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:05.103317 8344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:05.106362 8344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
format_stamp: "Formatted at 2026-09-26 23:24:05 on dist-test-slave-q5g8"
server_key: "e94751ddd51198bc8850c31ccc1240f7"
server_key_iv: "6e7964e2c397b026b0c01fcb74ef1ef9"
server_key_version: "encryptionkey@0"
I20260926 23:24:05.107152 8344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
format_stamp: "Formatted at 2026-09-26 23:24:05 on dist-test-slave-q5g8"
server_key: "e94751ddd51198bc8850c31ccc1240f7"
server_key_iv: "6e7964e2c397b026b0c01fcb74ef1ef9"
server_key_version: "encryptionkey@0"
I20260926 23:24:05.112442 8344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:24:05.116364 8359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:05.118002 8344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:05.118180 8344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
format_stamp: "Formatted at 2026-09-26 23:24:05 on dist-test-slave-q5g8"
server_key: "e94751ddd51198bc8850c31ccc1240f7"
server_key_iv: "6e7964e2c397b026b0c01fcb74ef1ef9"
server_key_version: "encryptionkey@0"
I20260926 23:24:05.118432 8344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:05.142490 8201 heartbeater.cc:503] Master 127.0.97.62:35845 was elected leader, sending a full tablet report...
I20260926 23:24:05.150664 8344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:05.155293 8344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:05.155542 8344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:05.158767 8344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:05.158905 8344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:05.159072 8344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:05.159149 8344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:24:05 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465045, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:24:05.185487 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.159847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35845 (local address 127.0.97.3:59845)
0926 23:24:05.160636 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:05.160718 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:05.162860 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:05.163670 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:05.163679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:05.164243 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:05.165046 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.165080 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.166966 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.166973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.169365 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.169385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.170561 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.170573 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.170947 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.171943 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:05.171972 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:05.175499 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:05.179614 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:05.179631 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:05.179655 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:05.180315 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:05.180941 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:05.180954 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:05.180962 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:05.181337 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:05.182217 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:05.182228 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:05.182710 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 23:24:05.183114 (+ 404us) client_negotiation.cc:240] Negotiation successful
0926 23:24:05.183742 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":334,"threads_started":1}
I20260926 23:24:05.186290 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.159103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59845 (local address 127.0.97.62:35845)
0926 23:24:05.161518 (+ 2415us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.161526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.161550 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:24:05.163174 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.163179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.163273 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.163410 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:05.165295 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.166771 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.169633 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.170250 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.170408 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.175783 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:05.175790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:05.175794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:05.175839 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:05.179447 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:05.180508 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:05.180518 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:05.180523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:05.180758 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:05.181561 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:05.181571 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:05.181577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:05.182044 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:05.182244 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.185627 (+ 3383us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.185905 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2277,"thread_start_us":2161,"threads_started":1}
I20260926 23:24:05.206970 8344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33507
I20260926 23:24:05.208629 8344 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:24:05.209257 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33507 every 8 connection(s)
I20260926 23:24:05.210534 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.192002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35845 (local address 127.0.97.3:35467)
0926 23:24:05.192985 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:05.193009 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:05.193224 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:05.193625 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:05.193634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:05.194615 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:05.195185 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.195209 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.197116 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.197121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.198304 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.198317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.199611 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.199622 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.199856 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.200648 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:05.200719 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:05.202378 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:05.207063 (+ 4685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:05.207069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:05.207073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:05.207564 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:05.208024 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:05.208028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:05.208031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:05.208166 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:05.208966 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:05.208972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:05.209209 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 23:24:05.209498 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 23:24:05.209974 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":560,"threads_started":1}
I20260926 23:24:05.211014 8480 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35845
I20260926 23:24:05.211268 8480 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:05.211589 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.191797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35467 (local address 127.0.97.62:35845)
0926 23:24:05.192015 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.192022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.193106 (+ 1084us) server_negotiation.cc:407] Connection header received
0926 23:24:05.193369 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.193375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.193468 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.193571 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:05.195449 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.196934 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.198470 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.199345 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.199530 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.202695 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:05.202705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:05.202714 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:05.202779 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:05.206887 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:05.207700 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:05.207705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:05.207709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:05.207872 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:05.208298 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:05.208304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:05.208307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:05.208742 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:05.208899 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.210999 (+ 2100us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.211252 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:24:05.212333 8480 heartbeater.cc:511] Master 127.0.97.62:35845 requested a full tablet report, sending...
I20260926 23:24:05.213794 7993 ts_manager.cc:194] Registered new tserver with Master: 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:05.214895 7993 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59845
I20260926 23:24:05.217440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 8344
I20260926 23:24:05.217617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:24:05.218047 388 external_mini_cluster.cc:1428] Setting key c36d7bf7ff3bb296a27ae936e6386add
I20260926 23:24:05.221101 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:24:05 dist-test-slave-q5g8 krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465045, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:24:05 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:05 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:24:05.272951 8367 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:05.262812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.0.97.62:35845)
0926 23:24:05.262987 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.262992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.263352 (+ 360us) server_negotiation.cc:407] Connection header received
0926 23:24:05.263505 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.263509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.263570 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.263659 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:05.265054 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.265766 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.266955 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.267341 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.267435 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.272561 (+ 5126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51752: BlockingRecv error: recv got EOF from 127.0.0.1:51752 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 23:24:05 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465045, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:24:05.290781 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.276326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.0.97.62:35845)
0926 23:24:05.276491 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.276495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.276877 (+ 382us) server_negotiation.cc:407] Connection header received
0926 23:24:05.277011 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.277015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.277075 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.277161 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:05.278973 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.279667 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.281163 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.281608 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.281703 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.285341 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:05.285346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:05.285350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:05.285390 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:05.288449 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:05.289224 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:05.289229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:05.289231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:05.289378 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:05.289699 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:05.289702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:05.289704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:05.289974 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:05.290179 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.290379 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.290544 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:24:05.294736 7993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:24:05.301784 7993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:24:05.329069 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.315008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38325 (local address 127.0.0.1:38350)
0926 23:24:05.317676 (+ 2668us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:05.317694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:05.317818 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:05.318917 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:05.318928 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:05.318968 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:05.319415 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.319430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.321820 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.321829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.325262 (+ 3433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.325287 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.328036 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.328041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.328267 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.328333 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 23:24:05.328459 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 23:24:05.328652 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2542,"thread_start_us":2406,"threads_started":1}
I20260926 23:24:05.329182 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.315418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.0.97.2:38325)
0926 23:24:05.316388 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.316401 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.317950 (+ 1549us) server_negotiation.cc:407] Connection header received
0926 23:24:05.318071 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.318078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.318374 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.318616 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:05.319642 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.321618 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.325531 (+ 3913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.327456 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.327580 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.327820 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.328653 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.328864 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":559,"threads_started":1}
I20260926 23:24:05.331839 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.314887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33507 (local address 127.0.0.1:57442)
0926 23:24:05.317266 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:05.317349 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:05.317590 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:05.320644 (+ 3054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:05.320652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:05.320701 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:05.321219 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.321233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.325193 (+ 3960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.325202 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.329177 (+ 3975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.329195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.331022 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.331028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.331196 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.331225 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 23:24:05.331297 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 23:24:05.331453 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1898,"mutex_wait_us":1,"thread_start_us":1708,"threads_started":1}
I20260926 23:24:05.332949 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.315156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.0.97.3:33507)
0926 23:24:05.319562 (+ 4406us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.319574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.319613 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:24:05.319720 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.319730 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.320018 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.320241 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:05.321506 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.324982 (+ 3476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.329392 (+ 4410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.330820 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.330948 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.332357 (+ 1409us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.332430 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.332602 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4199,"thread_start_us":4017,"threads_started":1}
I20260926 23:24:05.333371 8270 tablet_service.cc:1467] Processing CreateTablet for tablet 6a14f580f02244c9abcaab8185323226 (DEFAULT_TABLE table=test-table [id=6a2beb7fbd8a4cb4ab2be61057e89d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:05.335186 8270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a14f580f02244c9abcaab8185323226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:05.336652 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.318368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.0.97.1:43483)
0926 23:24:05.328583 (+ 10215us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.328592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.328618 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:24:05.328730 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.328738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.328989 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.329347 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:05.330119 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.331523 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.334471 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.335626 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.335775 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.335980 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.336197 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.336362 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10061,"thread_start_us":9912,"threads_started":1}
I20260926 23:24:05.336673 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.317884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43483 (local address 127.0.0.1:35876)
0926 23:24:05.327967 (+ 10083us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:05.327988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:05.328103 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:05.329539 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:05.329544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:05.329575 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:05.329924 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.329937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.332136 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.332140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.334307 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.334322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.335834 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.335840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.336005 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.336037 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:24:05.336156 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 23:24:05.336285 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9932,"thread_start_us":11561,"threads_started":1}
I20260926 23:24:05.336901 8410 tablet_service.cc:1467] Processing CreateTablet for tablet 6a14f580f02244c9abcaab8185323226 (DEFAULT_TABLE table=test-table [id=6a2beb7fbd8a4cb4ab2be61057e89d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:05.338339 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a14f580f02244c9abcaab8185323226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:05.340456 8131 tablet_service.cc:1467] Processing CreateTablet for tablet 6a14f580f02244c9abcaab8185323226 (DEFAULT_TABLE table=test-table [id=6a2beb7fbd8a4cb4ab2be61057e89d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:05.341763 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a14f580f02244c9abcaab8185323226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:05.351195 8504 tablet_bootstrap.cc:492] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Bootstrap starting.
I20260926 23:24:05.353251 8505 tablet_bootstrap.cc:492] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Bootstrap starting.
I20260926 23:24:05.356169 8505 tablet_bootstrap.cc:654] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:05.356901 8504 tablet_bootstrap.cc:654] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:05.357362 8505 log.cc:824] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:05.358459 8504 log.cc:824] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:05.359874 8506 tablet_bootstrap.cc:492] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Bootstrap starting.
I20260926 23:24:05.362071 8504 tablet_bootstrap.cc:492] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: No bootstrap required, opened a new log
I20260926 23:24:05.362373 8504 ts_tablet_manager.cc:1397] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260926 23:24:05.364163 8506 tablet_bootstrap.cc:654] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:05.365414 8506 log.cc:824] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:05.368019 8505 tablet_bootstrap.cc:492] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: No bootstrap required, opened a new log
I20260926 23:24:05.368347 8505 ts_tablet_manager.cc:1397] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260926 23:24:05.370702 8506 tablet_bootstrap.cc:492] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: No bootstrap required, opened a new log
I20260926 23:24:05.370980 8506 ts_tablet_manager.cc:1397] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 23:24:05.373426 8504 raft_consensus.cc:348] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.373891 8504 raft_consensus.cc:374] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:05.374006 8504 raft_consensus.cc:729] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c553c6025f1a42e3a050f2e7ea31cde2, State: Initialized, Role: FOLLOWER
I20260926 23:24:05.374686 8504 consensus_queue.cc:260] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.375941 8340 heartbeater.cc:503] Master 127.0.97.62:35845 was elected leader, sending a full tablet report...
I20260926 23:24:05.376734 8504 ts_tablet_manager.cc:1428] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260926 23:24:05.376372 8505 raft_consensus.cc:348] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.377089 8505 raft_consensus.cc:374] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:05.377194 8505 raft_consensus.cc:729] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdcad86a04f4b0388c3537b07ccb3a1, State: Initialized, Role: FOLLOWER
I20260926 23:24:05.378144 8505 consensus_queue.cc:260] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.381340 8480 heartbeater.cc:503] Master 127.0.97.62:35845 was elected leader, sending a full tablet report...
I20260926 23:24:05.382062 8505 ts_tablet_manager.cc:1428] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 23:24:05.380947 8506 raft_consensus.cc:348] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.382544 8506 raft_consensus.cc:374] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:05.382668 8506 raft_consensus.cc:729] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Initialized, Role: FOLLOWER
I20260926 23:24:05.383255 8506 consensus_queue.cc:260] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.384938 8506 ts_tablet_manager.cc:1428] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260926 23:24:05.421020 8341 tablet.cc:2369] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:24:05.460659 8481 tablet.cc:2369] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:24:05.634423 8202 tablet.cc:2369] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:24:05.777264 8510 raft_consensus.cc:482] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:05.777593 8510 raft_consensus.cc:504] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.779304 8510 leader_election.cc:288] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507), f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483)
I20260926 23:24:05.786053 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.779676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34571 (local address 127.0.97.3:33507)
0926 23:24:05.779888 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.779894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.780787 (+ 893us) server_negotiation.cc:407] Connection header received
0926 23:24:05.781081 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.781087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.781157 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.781253 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:05.781819 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.782504 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.784265 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.785097 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.785244 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.785351 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.785661 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.785781 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:24:05.786187 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.779809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50215 (local address 127.0.97.1:43483)
0926 23:24:05.780043 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.780051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.780956 (+ 905us) server_negotiation.cc:407] Connection header received
0926 23:24:05.781163 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.781168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.781258 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.781357 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:05.782015 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.782702 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.784400 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.785247 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.785421 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.785528 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.785817 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.785938 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:24:05.786214 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.779798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33507 (local address 127.0.97.2:34571)
0926 23:24:05.780640 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:05.780664 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:05.780925 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:05.781279 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:05.781283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:05.781320 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:05.781673 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.781681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.782640 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.782647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.784112 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.784123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.785297 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.785302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.785481 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.785523 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:24:05.785627 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:24:05.785754 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":3,"spinlock_wait_cycles":25344,"thread_start_us":443,"threads_started":1}
I20260926 23:24:05.786399 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.779844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43483 (local address 127.0.97.2:50215)
0926 23:24:05.780861 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:05.780880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:05.781019 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:05.781370 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:05.781378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:05.781481 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:05.781862 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.781870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.782835 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.782843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.784244 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:05.784255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.785486 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.785492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:05.785664 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.785686 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:24:05.785793 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 23:24:05.785920 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":18,"spinlock_wait_cycles":31488,"thread_start_us":586,"threads_started":1}
I20260926 23:24:05.787173 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a14f580f02244c9abcaab8185323226" candidate_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" is_pre_election: true
I20260926 23:24:05.787333 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a14f580f02244c9abcaab8185323226" candidate_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" is_pre_election: true
I20260926 23:24:05.787680 8430 raft_consensus.cc:2454] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c553c6025f1a42e3a050f2e7ea31cde2 in term 0.
I20260926 23:24:05.787895 8156 raft_consensus.cc:2454] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c553c6025f1a42e3a050f2e7ea31cde2 in term 0.
I20260926 23:24:05.788287 8224 leader_election.cc:302] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, c553c6025f1a42e3a050f2e7ea31cde2; no voters:
I20260926 23:24:05.788764 8510 raft_consensus.cc:2781] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:24:05.788914 8510 raft_consensus.cc:482] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:24:05.789034 8510 raft_consensus.cc:3037] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:05.791294 8510 raft_consensus.cc:504] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.791920 8510 leader_election.cc:288] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [CANDIDATE]: Term 1 election: Requested vote from peers 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507), f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483)
I20260926 23:24:05.792367 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a14f580f02244c9abcaab8185323226" candidate_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
I20260926 23:24:05.792429 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a14f580f02244c9abcaab8185323226" candidate_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338"
I20260926 23:24:05.792616 8156 raft_consensus.cc:3037] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:05.792606 8430 raft_consensus.cc:3037] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:05.795722 8430 raft_consensus.cc:2454] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c553c6025f1a42e3a050f2e7ea31cde2 in term 1.
I20260926 23:24:05.795722 8156 raft_consensus.cc:2454] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c553c6025f1a42e3a050f2e7ea31cde2 in term 1.
I20260926 23:24:05.796124 8224 leader_election.cc:302] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, c553c6025f1a42e3a050f2e7ea31cde2; no voters:
I20260926 23:24:05.796465 8510 raft_consensus.cc:2781] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:05.796975 8510 raft_consensus.cc:686] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 LEADER]: Becoming Leader. State: Replica: c553c6025f1a42e3a050f2e7ea31cde2, State: Running, Role: LEADER
I20260926 23:24:05.797520 8510 consensus_queue.cc:237] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } }
I20260926 23:24:05.802276 7994 catalog_manager.cc:5495] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } health_report { overall_health: UNKNOWN } } }
I20260926 23:24:05.892395 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:05.886442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.0.97.2:38325)
0926 23:24:05.886997 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:05.887002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:05.887020 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:24:05.887252 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:05.887257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:05.887365 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:05.887531 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:24:05.888027 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.889031 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.890398 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:05.890795 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:05.890903 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:05.891264 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:24:05.891871 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:24:05.891985 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:05.892073 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 23:24:05.892139 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":345,"threads_started":1}
I20260926 23:24:05.900137 8156 raft_consensus.cc:1261] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Refusing update from remote peer c553c6025f1a42e3a050f2e7ea31cde2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:05.900223 8430 raft_consensus.cc:1261] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Refusing update from remote peer c553c6025f1a42e3a050f2e7ea31cde2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:05.901203 8515 consensus_queue.cc:1035] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:05.901597 8510 consensus_queue.cc:1035] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:05.916778 8520 mvcc.cc:205] Tried to move back new op lower bound from 7333744827998023680 to 7333744827590639616. Current Snapshot: MvccSnapshot[applied={T|T < 7333744827998023680 or (T in {7333744827998023680})}]
I20260926 23:24:05.926524 7993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:05.926810 7993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:05.930332 7993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Sending DeleteTablet for 3 replicas of tablet 6a14f580f02244c9abcaab8185323226
I20260926 23:24:05.931504 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a14f580f02244c9abcaab8185323226 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57442
I20260926 23:24:05.931643 8131 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a14f580f02244c9abcaab8185323226 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35876
I20260926 23:24:05.932189 8270 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a14f580f02244c9abcaab8185323226 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38350
I20260926 23:24:05.935228 7993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:24:05.936473 7993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:24:05.948261 8409 tablet_service.cc:1467] Processing CreateTablet for tablet dfcb4c78e1bf482fabaad233a828d148 (DEFAULT_TABLE table=test-table [id=a511fc17ca4d4adc8825c37824502d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:05.948912 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcb4c78e1bf482fabaad233a828d148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:05.948860 8130 tablet_service.cc:1467] Processing CreateTablet for tablet dfcb4c78e1bf482fabaad233a828d148 (DEFAULT_TABLE table=test-table [id=a511fc17ca4d4adc8825c37824502d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:05.949329 8130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcb4c78e1bf482fabaad233a828d148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:05.953217 8269 tablet_service.cc:1467] Processing CreateTablet for tablet dfcb4c78e1bf482fabaad233a828d148 (DEFAULT_TABLE table=test-table [id=a511fc17ca4d4adc8825c37824502d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:05.953835 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcb4c78e1bf482fabaad233a828d148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:05.964375 8534 tablet_replica.cc:330] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: stopping tablet replica
I20260926 23:24:05.964624 8535 tablet_bootstrap.cc:492] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: Bootstrap starting.
I20260926 23:24:05.964916 8534 raft_consensus.cc:2229] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:24:05.966766 8535 tablet_bootstrap.cc:654] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:05.967413 8536 tablet_bootstrap.cc:492] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: Bootstrap starting.
I20260926 23:24:05.968596 8534 raft_consensus.cc:2258] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:05.970608 8536 tablet_bootstrap.cc:654] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:05.972075 8532 tablet_replica.cc:330] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: stopping tablet replica
I20260926 23:24:05.973404 8537 tablet_bootstrap.cc:492] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: Bootstrap starting.
I20260926 23:24:05.974328 8532 raft_consensus.cc:2229] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:05.974647 8534 ts_tablet_manager.cc:1905] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:05.974892 8533 tablet_replica.cc:330] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: stopping tablet replica
I20260926 23:24:05.975330 8533 raft_consensus.cc:2229] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:05.975888 8533 raft_consensus.cc:2258] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:05.976101 8532 raft_consensus.cc:2258] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:05.977887 8533 ts_tablet_manager.cc:1905] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:05.978029 8535 tablet_bootstrap.cc:492] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: No bootstrap required, opened a new log
I20260926 23:24:05.978181 8535 ts_tablet_manager.cc:1397] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20260926 23:24:05.978583 8532 ts_tablet_manager.cc:1905] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:05.981252 8536 tablet_bootstrap.cc:492] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: No bootstrap required, opened a new log
I20260926 23:24:05.981205 8535 raft_consensus.cc:348] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:05.981406 8536 ts_tablet_manager.cc:1397] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260926 23:24:05.981460 8535 raft_consensus.cc:374] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:05.981566 8535 raft_consensus.cc:729] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdcad86a04f4b0388c3537b07ccb3a1, State: Initialized, Role: FOLLOWER
I20260926 23:24:05.981850 8535 consensus_queue.cc:260] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:05.982452 8535 ts_tablet_manager.cc:1428] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260926 23:24:05.985445 8533 ts_tablet_manager.cc:1918] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:05.985610 8533 log.cc:1196] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/6a14f580f02244c9abcaab8185323226
I20260926 23:24:05.986020 8532 ts_tablet_manager.cc:1918] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:05.986135 8533 ts_tablet_manager.cc:1939] T 6a14f580f02244c9abcaab8185323226 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting consensus metadata
I20260926 23:24:05.986302 8532 log.cc:1196] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/6a14f580f02244c9abcaab8185323226
I20260926 23:24:05.986936 8532 ts_tablet_manager.cc:1939] T 6a14f580f02244c9abcaab8185323226 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting consensus metadata
I20260926 23:24:05.987521 8536 raft_consensus.cc:348] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:05.987771 8536 raft_consensus.cc:374] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:05.987874 8536 raft_consensus.cc:729] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Initialized, Role: FOLLOWER
I20260926 23:24:05.987965 7978 catalog_manager.cc:4841] TS f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483): tablet 6a14f580f02244c9abcaab8185323226 (table test-table [id=6a2beb7fbd8a4cb4ab2be61057e89d2f]) successfully deleted
I20260926 23:24:05.988899 7981 catalog_manager.cc:4841] TS 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507): tablet 6a14f580f02244c9abcaab8185323226 (table test-table [id=6a2beb7fbd8a4cb4ab2be61057e89d2f]) successfully deleted
I20260926 23:24:05.988894 8536 consensus_queue.cc:260] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:05.989552 8536 ts_tablet_manager.cc:1428] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260926 23:24:05.989962 8537 tablet_bootstrap.cc:654] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:05.992261 8537 tablet_bootstrap.cc:492] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: No bootstrap required, opened a new log
I20260926 23:24:05.992420 8537 ts_tablet_manager.cc:1397] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20260926 23:24:05.993328 8537 raft_consensus.cc:348] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:05.993592 8537 raft_consensus.cc:374] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:05.993705 8537 raft_consensus.cc:729] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c553c6025f1a42e3a050f2e7ea31cde2, State: Initialized, Role: FOLLOWER
I20260926 23:24:05.994014 8537 consensus_queue.cc:260] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:05.994750 8537 ts_tablet_manager.cc:1428] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 23:24:05.995894 8534 ts_tablet_manager.cc:1918] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:05.996081 8534 log.cc:1196] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/6a14f580f02244c9abcaab8185323226
I20260926 23:24:05.996650 8534 ts_tablet_manager.cc:1939] T 6a14f580f02244c9abcaab8185323226 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting consensus metadata
I20260926 23:24:05.998183 7978 catalog_manager.cc:4841] TS c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325): tablet 6a14f580f02244c9abcaab8185323226 (table test-table [id=6a2beb7fbd8a4cb4ab2be61057e89d2f]) successfully deleted
I20260926 23:24:06.089711 8521 raft_consensus.cc:482] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:06.089974 8521 raft_consensus.cc:504] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.091600 8521 leader_election.cc:288] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.101069 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.091976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38325 (local address 127.0.97.1:42037)
0926 23:24:06.093057 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:06.093078 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:06.093336 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:06.093789 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:06.093793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:06.093813 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:06.094344 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.094360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.095892 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.095898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.098588 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.098604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.100225 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.100230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.100344 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.100370 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 23:24:06.100481 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:24:06.100609 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":269,"thread_start_us":1220,"threads_started":1}
I20260926 23:24:06.101146 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.092005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42037 (local address 127.0.97.2:38325)
0926 23:24:06.092313 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.092319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.093174 (+ 855us) server_negotiation.cc:407] Connection header received
0926 23:24:06.093565 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.093571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.093658 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.093755 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:06.094522 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.095707 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.098752 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.100027 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.100170 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.100273 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.100744 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.100900 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260926 23:24:06.102622 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.092265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40347 (local address 127.0.97.3:33507)
0926 23:24:06.092470 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.092479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.093577 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 23:24:06.093764 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.093770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.093846 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.093949 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:06.094692 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.095784 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.100355 (+ 4571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.101631 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.101754 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.101859 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.102221 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.102346 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:24:06.102696 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.092022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33507 (local address 127.0.97.1:40347)
0926 23:24:06.093494 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:06.093513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:06.093613 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:06.093987 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:06.093992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:06.094019 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:06.094515 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.094527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.095917 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.095923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.098600 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.098609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.101869 (+ 3260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.101879 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.102047 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.102070 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:24:06.102176 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:24:06.102295 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":156,"thread_start_us":771,"threads_started":1}
I20260926 23:24:06.103444 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcb4c78e1bf482fabaad233a828d148" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" is_pre_election: true
I20260926 23:24:06.103758 8430 raft_consensus.cc:2454] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.103616 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcb4c78e1bf482fabaad233a828d148" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" is_pre_election: true
I20260926 23:24:06.104041 8295 raft_consensus.cc:2454] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.104285 8084 leader_election.cc:302] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.104719 8521 raft_consensus.cc:2781] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:24:06.104835 8521 raft_consensus.cc:482] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:24:06.104923 8521 raft_consensus.cc:3037] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.107121 8521 raft_consensus.cc:504] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.107887 8521 leader_election.cc:288] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Requested vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.108279 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcb4c78e1bf482fabaad233a828d148" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2"
I20260926 23:24:06.108476 8295 raft_consensus.cc:3037] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.108486 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcb4c78e1bf482fabaad233a828d148" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
I20260926 23:24:06.108752 8430 raft_consensus.cc:3037] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.110646 8430 raft_consensus.cc:2454] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.111004 8084 leader_election.cc:302] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.111280 8521 raft_consensus.cc:2781] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:06.111725 8295 raft_consensus.cc:2454] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.111829 8521 raft_consensus.cc:686] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 LEADER]: Becoming Leader. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Running, Role: LEADER
I20260926 23:24:06.112530 8521 consensus_queue.cc:237] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.116178 7994 catalog_manager.cc:5495] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 reported cstate change: term changed from 0 to 1, leader changed from <none> to f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } health_report { overall_health: UNKNOWN } } }
Sep 26 23:24:06 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:24:06.195363 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.171506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.97.62:35845)
0926 23:24:06.172608 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.172613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.172638 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:24:06.172676 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.172696 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.172787 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.172895 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:06.174147 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.175031 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.176296 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.176753 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.176892 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.189878 (+ 12986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:06.189883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:06.189887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:06.189917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:06.192592 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:06.193484 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:06.193488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:06.193491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:06.193610 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:06.194015 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:06.194018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:06.194020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:06.194289 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:06.194394 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.194903 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.195069 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":835,"threads_started":1}
I20260926 23:24:06.195808 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.171507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35845 (local address 127.0.0.1:51782)
0926 23:24:06.172286 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:06.172313 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:06.172489 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:06.172919 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:06.172923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:06.173685 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:06.173995 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.174005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.175158 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.175165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.176148 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.176159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.176912 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.176918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.177117 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.186833 (+ 9716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:06.186856 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:06.189627 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:06.192765 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:06.192777 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:06.192785 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:06.193367 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:06.193719 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:06.193727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:06.193731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:06.193906 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:06.194428 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:06.194437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:06.194638 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 23:24:06.194840 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 23:24:06.195244 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":80,"thread_start_us":451,"threads_started":1}
I20260926 23:24:06.201406 7994 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51782:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:24:06.202513 7994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:24:06.210645 8131 tablet_service.cc:1467] Processing CreateTablet for tablet 0b2688bea1994d0e9b870c79103ea2ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83c8403335b048ee8d073f2131af5fec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:24:06.210801 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 0b2688bea1994d0e9b870c79103ea2ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83c8403335b048ee8d073f2131af5fec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:24:06.211292 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2688bea1994d0e9b870c79103ea2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.211352 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2688bea1994d0e9b870c79103ea2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.217685 8410 tablet_service.cc:1467] Processing CreateTablet for tablet 0b2688bea1994d0e9b870c79103ea2ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83c8403335b048ee8d073f2131af5fec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:24:06.218497 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2688bea1994d0e9b870c79103ea2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.219274 8537 tablet_bootstrap.cc:492] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: Bootstrap starting.
I20260926 23:24:06.221297 8537 tablet_bootstrap.cc:654] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.226833 8535 tablet_bootstrap.cc:492] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: Bootstrap starting.
I20260926 23:24:06.229091 8535 tablet_bootstrap.cc:654] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.229149 8536 tablet_bootstrap.cc:492] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: Bootstrap starting.
I20260926 23:24:06.231045 8536 tablet_bootstrap.cc:654] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.231842 8535 tablet_bootstrap.cc:492] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: No bootstrap required, opened a new log
I20260926 23:24:06.232046 8535 ts_tablet_manager.cc:1397] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 23:24:06.232144 8537 tablet_bootstrap.cc:492] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: No bootstrap required, opened a new log
I20260926 23:24:06.232383 8537 ts_tablet_manager.cc:1397] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260926 23:24:06.232930 8535 raft_consensus.cc:348] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.233222 8535 raft_consensus.cc:374] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.233302 8535 raft_consensus.cc:729] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdcad86a04f4b0388c3537b07ccb3a1, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.233552 8535 consensus_queue.cc:260] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.233166 8537 raft_consensus.cc:348] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.233847 8537 raft_consensus.cc:374] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.233944 8537 raft_consensus.cc:729] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c553c6025f1a42e3a050f2e7ea31cde2, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.234081 8522 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.234246 8537 consensus_queue.cc:260] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.234529 8522 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:24:06.234752 8510 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.235028 8510 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:24:06.235594 8536 tablet_bootstrap.cc:492] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: No bootstrap required, opened a new log
I20260926 23:24:06.235859 8536 ts_tablet_manager.cc:1397] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260926 23:24:06.236712 8536 raft_consensus.cc:348] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.237015 8536 raft_consensus.cc:374] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.237103 8536 raft_consensus.cc:729] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.237401 8536 consensus_queue.cc:260] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.237838 8521 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.238061 8536 ts_tablet_manager.cc:1428] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:24:06.238132 8521 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:24:06.234084 8535 ts_tablet_manager.cc:1428] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 23:24:06.239020 8537 ts_tablet_manager.cc:1428] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260926 23:24:06.268976 8521 raft_consensus.cc:482] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:06.269240 8521 raft_consensus.cc:504] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.270016 8521 leader_election.cc:288] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.270607 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2688bea1994d0e9b870c79103ea2ad" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" is_pre_election: true
I20260926 23:24:06.270617 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2688bea1994d0e9b870c79103ea2ad" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" is_pre_election: true
I20260926 23:24:06.270896 8295 raft_consensus.cc:2454] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.270886 8430 raft_consensus.cc:2454] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.271332 8084 leader_election.cc:302] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.271618 8521 raft_consensus.cc:2781] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:24:06.271728 8521 raft_consensus.cc:482] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:24:06.271852 8521 raft_consensus.cc:3037] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.275043 8521 raft_consensus.cc:504] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.275828 8521 leader_election.cc:288] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Requested vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.276304 8295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2688bea1994d0e9b870c79103ea2ad" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2"
I20260926 23:24:06.276350 8430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2688bea1994d0e9b870c79103ea2ad" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
I20260926 23:24:06.276569 8295 raft_consensus.cc:3037] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.276562 8430 raft_consensus.cc:3037] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.279601 8430 raft_consensus.cc:2454] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.279758 8295 raft_consensus.cc:2454] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.280227 8081 leader_election.cc:302] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, c553c6025f1a42e3a050f2e7ea31cde2, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.280586 8521 raft_consensus.cc:2781] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:06.280783 8521 raft_consensus.cc:686] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 LEADER]: Becoming Leader. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Running, Role: LEADER
I20260926 23:24:06.281090 8521 consensus_queue.cc:237] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.283020 8543 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: TxnStatusTablet state changed. Reason: New leader f583e7ac98ff4f2eb80d1d783ad3b338. Latest consensus state: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.283427 8543 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:24:06.284236 8556 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:24:06.284485 8556 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:24:06.285480 7993 catalog_manager.cc:5495] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 reported cstate change: term changed from 0 to 1, leader changed from <none> to f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } health_report { overall_health: UNKNOWN } } }
I20260926 23:24:06.311645 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.305124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43483 (local address 127.0.0.1:35884)
0926 23:24:06.305556 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:06.305582 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:06.305758 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:06.306750 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:06.306754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:06.306767 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:24:06.306954 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.306970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.308134 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.308142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.309211 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.309222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.309935 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.309941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.310128 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.310181 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:24:06.311089 (+ 908us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:24:06.311109 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 23:24:06.311195 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:24:06.311349 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260926 23:24:06.311681 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.305116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.0.97.1:43483)
0926 23:24:06.306390 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.306395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.306410 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:24:06.306486 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.306490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.306567 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.306729 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:24:06.307119 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.308002 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.309354 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.309786 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.309922 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.310408 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:24:06.310980 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:24:06.311135 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.311216 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.311295 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":1002,"threads_started":1}
I20260926 23:24:06.315739 8430 raft_consensus.cc:1261] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.315943 8295 raft_consensus.cc:1261] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.316501 8521 consensus_queue.cc:1035] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:06.317258 8521 consensus_queue.cc:1035] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:06.336974 8510 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: TxnStatusTablet state changed. Reason: New leader f583e7ac98ff4f2eb80d1d783ad3b338. Latest consensus state: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.337280 8510 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:24:06.337894 8521 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.338207 8521 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:24:06.339025 8521 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.339210 8521 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P f583e7ac98ff4f2eb80d1d783ad3b338: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:24:06.342269 8522 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: TxnStatusTablet state changed. Reason: New leader f583e7ac98ff4f2eb80d1d783ad3b338. Latest consensus state: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.342561 8522 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:24:06.344496 8510 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.344823 8510 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P c553c6025f1a42e3a050f2e7ea31cde2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:24:06.350679 8524 mvcc.cc:205] Tried to move back new op lower bound from 7333744829701787648 to 7333744829569978368. Current Snapshot: MvccSnapshot[applied={T|T < 7333744829701787648}]
I20260926 23:24:06.352305 8522 tablet_replica.cc:439] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } } }
I20260926 23:24:06.352555 8522 tablet_replica.cc:442] T 0b2688bea1994d0e9b870c79103ea2ad P 8fdcad86a04f4b0388c3537b07ccb3a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:24:06.358533 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.348279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.0.97.1:43483)
0926 23:24:06.348508 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.348513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.348530 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:24:06.348590 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.348596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.348716 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.349285 (+ 569us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:24:06.349423 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.350447 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.356395 (+ 5948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.357076 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.357272 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.357725 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:24:06.357856 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:24:06.357949 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.358104 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.358181 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 23:24:06.375699 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.369509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43539 (local address 127.0.97.1:43483)
0926 23:24:06.369723 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.369731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.370421 (+ 690us) server_negotiation.cc:407] Connection header received
0926 23:24:06.370632 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.370637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.370727 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.370816 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:24:06.371288 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.372025 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.373659 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.374170 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.374308 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.374809 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:24:06.374954 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:24:06.375034 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.375164 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.375300 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":60}
I20260926 23:24:06.376181 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.369554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43483 (local address 127.0.97.1:43539)
0926 23:24:06.370321 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:06.370338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:06.370508 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:06.370832 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:06.370836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:06.370846 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:24:06.371149 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.371159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.372186 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.372191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.373530 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.373543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.374363 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.374369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.374564 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.374642 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:24:06.375032 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:24:06.375052 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 23:24:06.375148 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:24:06.375352 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":38,"thread_start_us":437,"threads_started":1}
I20260926 23:24:06.386137 8295 raft_consensus.cc:1261] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.386189 8430 raft_consensus.cc:1261] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.386780 8521 consensus_queue.cc:1035] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:06.387301 8521 consensus_queue.cc:1035] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:06.393136 8523 mvcc.cc:205] Tried to move back new op lower bound from 7333744829990510592 to 7333744828881313792. Current Snapshot: MvccSnapshot[applied={T|T < 7333744829990510592}]
I20260926 23:24:06.462438 7991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:06.462740 7991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:06.465798 7991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Sending DeleteTablet for 3 replicas of tablet dfcb4c78e1bf482fabaad233a828d148
I20260926 23:24:06.466642 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcb4c78e1bf482fabaad233a828d148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38350
I20260926 23:24:06.466747 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcb4c78e1bf482fabaad233a828d148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57442
I20260926 23:24:06.466954 8534 tablet_replica.cc:330] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: stopping tablet replica
I20260926 23:24:06.467163 8534 raft_consensus.cc:2229] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:06.467341 8534 raft_consensus.cc:2258] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:06.467365 8131 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcb4c78e1bf482fabaad233a828d148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35876
I20260926 23:24:06.467597 8532 tablet_replica.cc:330] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: stopping tablet replica
I20260926 23:24:06.467774 8533 tablet_replica.cc:330] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: stopping tablet replica
I20260926 23:24:06.467972 8533 raft_consensus.cc:2229] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:24:06.467995 8532 raft_consensus.cc:2229] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:06.468168 8532 raft_consensus.cc:2258] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:06.468482 8533 raft_consensus.cc:2258] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:06.469729 8534 ts_tablet_manager.cc:1905] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:06.469722 8532 ts_tablet_manager.cc:1905] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:06.476213 8533 ts_tablet_manager.cc:1905] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:06.479458 8532 ts_tablet_manager.cc:1918] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:24:06.479619 8532 log.cc:1196] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/dfcb4c78e1bf482fabaad233a828d148
I20260926 23:24:06.480019 8532 ts_tablet_manager.cc:1939] T dfcb4c78e1bf482fabaad233a828d148 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting consensus metadata
I20260926 23:24:06.482430 7981 catalog_manager.cc:4841] TS 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507): tablet dfcb4c78e1bf482fabaad233a828d148 (table test-table [id=a511fc17ca4d4adc8825c37824502d6f]) successfully deleted
I20260926 23:24:06.484593 8534 ts_tablet_manager.cc:1918] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:24:06.484858 8534 log.cc:1196] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/dfcb4c78e1bf482fabaad233a828d148
I20260926 23:24:06.485332 8534 ts_tablet_manager.cc:1939] T dfcb4c78e1bf482fabaad233a828d148 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting consensus metadata
I20260926 23:24:06.486256 8533 ts_tablet_manager.cc:1918] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:24:06.486426 8533 log.cc:1196] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/dfcb4c78e1bf482fabaad233a828d148
I20260926 23:24:06.486569 7978 catalog_manager.cc:4841] TS c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325): tablet dfcb4c78e1bf482fabaad233a828d148 (table test-table [id=a511fc17ca4d4adc8825c37824502d6f]) successfully deleted
I20260926 23:24:06.486860 8533 ts_tablet_manager.cc:1939] T dfcb4c78e1bf482fabaad233a828d148 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting consensus metadata
I20260926 23:24:06.488092 7978 catalog_manager.cc:4841] TS f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483): tablet dfcb4c78e1bf482fabaad233a828d148 (table test-table [id=a511fc17ca4d4adc8825c37824502d6f]) successfully deleted
Sep 26 23:24:06 dist-test-slave-q5g8 krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:24:06 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:06 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:24:06.538236 8551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:06.527297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.0.97.62:35845)
0926 23:24:06.527505 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.527512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.527901 (+ 389us) server_negotiation.cc:407] Connection header received
0926 23:24:06.528054 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.528060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.528159 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.528284 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:06.529604 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.530692 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.531913 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.532564 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.532733 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.537982 (+ 5249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51790: BlockingRecv error: recv got EOF from 127.0.0.1:51790 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 23:24:06 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:24:06.555119 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.540953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.0.97.62:35845)
0926 23:24:06.541132 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.541138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.541627 (+ 489us) server_negotiation.cc:407] Connection header received
0926 23:24:06.541788 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.541798 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.541858 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.541938 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:06.543449 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.544164 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.545527 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.546071 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.546204 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.550049 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:06.550053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:06.550056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:06.550088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:06.552545 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:06.553195 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:06.553205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:06.553207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:06.553335 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:06.553717 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:06.553727 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:06.553729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:06.553992 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:06.554255 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.554556 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.554778 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:24:06.559106 7991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:24:06.559727 7991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:24:06.566442 8131 tablet_service.cc:1467] Processing CreateTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff (DEFAULT_TABLE table=test-table [id=621c3f96d11048489bee21b97372a521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:06.566466 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff (DEFAULT_TABLE table=test-table [id=621c3f96d11048489bee21b97372a521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:06.566830 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e93d79e995b4fd3b8fda1be65a864ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.566890 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e93d79e995b4fd3b8fda1be65a864ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.569769 8410 tablet_service.cc:1467] Processing CreateTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff (DEFAULT_TABLE table=test-table [id=621c3f96d11048489bee21b97372a521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:06.570199 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e93d79e995b4fd3b8fda1be65a864ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.572736 8536 tablet_bootstrap.cc:492] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: Bootstrap starting.
I20260926 23:24:06.573349 8537 tablet_bootstrap.cc:492] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: Bootstrap starting.
I20260926 23:24:06.574519 8536 tablet_bootstrap.cc:654] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.574878 8537 tablet_bootstrap.cc:654] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.576617 8535 tablet_bootstrap.cc:492] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: Bootstrap starting.
I20260926 23:24:06.578536 8535 tablet_bootstrap.cc:654] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.579710 8537 tablet_bootstrap.cc:492] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: No bootstrap required, opened a new log
I20260926 23:24:06.579876 8537 ts_tablet_manager.cc:1397] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 23:24:06.579990 8536 tablet_bootstrap.cc:492] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: No bootstrap required, opened a new log
I20260926 23:24:06.580163 8536 ts_tablet_manager.cc:1397] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260926 23:24:06.580643 8537 raft_consensus.cc:348] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.580936 8537 raft_consensus.cc:374] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.581034 8537 raft_consensus.cc:729] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c553c6025f1a42e3a050f2e7ea31cde2, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.581270 8537 consensus_queue.cc:260] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.581125 8536 raft_consensus.cc:348] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.581527 8536 raft_consensus.cc:374] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.581655 8536 raft_consensus.cc:729] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.581815 8537 ts_tablet_manager.cc:1428] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:24:06.581972 8536 consensus_queue.cc:260] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.583415 8536 ts_tablet_manager.cc:1428] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:06.592600 8535 tablet_bootstrap.cc:492] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: No bootstrap required, opened a new log
I20260926 23:24:06.592837 8535 ts_tablet_manager.cc:1397] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.007s
I20260926 23:24:06.593623 8535 raft_consensus.cc:348] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.593906 8535 raft_consensus.cc:374] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.594007 8535 raft_consensus.cc:729] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdcad86a04f4b0388c3537b07ccb3a1, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.594257 8535 consensus_queue.cc:260] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.594920 8535 ts_tablet_manager.cc:1428] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:24:06.771185 8522 raft_consensus.cc:482] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:06.771416 8522 raft_consensus.cc:504] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.773404 8522 leader_election.cc:288] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483), c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325)
I20260926 23:24:06.774121 8565 raft_consensus.cc:482] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:06.774330 8565 raft_consensus.cc:504] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.775269 8565 leader_election.cc:288] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.776118 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e93d79e995b4fd3b8fda1be65a864ff" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" is_pre_election: true
I20260926 23:24:06.776437 8429 raft_consensus.cc:2454] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.777071 8084 leader_election.cc:302] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.778028 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e93d79e995b4fd3b8fda1be65a864ff" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" is_pre_election: true
I20260926 23:24:06.778339 8293 raft_consensus.cc:2454] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.779130 8565 raft_consensus.cc:2781] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:24:06.779296 8565 raft_consensus.cc:482] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:24:06.779425 8565 raft_consensus.cc:3037] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.782933 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.773832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46031 (local address 127.0.97.1:43483)
0926 23:24:06.774077 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.774087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.774625 (+ 538us) server_negotiation.cc:407] Connection header received
0926 23:24:06.774981 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.774987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.775069 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.775132 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:06.775885 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.777232 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.780274 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.781710 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.781841 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.781946 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.782510 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.782652 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:24:06.783756 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.773629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43483 (local address 127.0.97.3:46031)
0926 23:24:06.774524 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:06.774544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:06.774815 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:06.775324 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:06.775329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:06.775411 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:06.775711 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.775723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.777479 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.777485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.780046 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.780062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.782099 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.782105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.782299 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.782334 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:24:06.783245 (+ 911us) client_negotiation.cc:240] Negotiation successful
0926 23:24:06.783392 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":572,"threads_started":1}
I20260926 23:24:06.783605 8565 raft_consensus.cc:504] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.784994 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e93d79e995b4fd3b8fda1be65a864ff" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2"
I20260926 23:24:06.785243 8293 raft_consensus.cc:3037] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.785221 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e93d79e995b4fd3b8fda1be65a864ff" candidate_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" is_pre_election: true
I20260926 23:24:06.785573 8156 raft_consensus.cc:2379] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8fdcad86a04f4b0388c3537b07ccb3a1 in current term 1: Already voted for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in this term.
I20260926 23:24:06.786500 8565 leader_election.cc:288] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Requested vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.787205 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e93d79e995b4fd3b8fda1be65a864ff" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
I20260926 23:24:06.787456 8429 raft_consensus.cc:3037] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.788578 8293 raft_consensus.cc:2454] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.789058 8081 leader_election.cc:302] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c553c6025f1a42e3a050f2e7ea31cde2, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.789512 8565 raft_consensus.cc:2781] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:06.790052 8565 raft_consensus.cc:686] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 LEADER]: Becoming Leader. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Running, Role: LEADER
I20260926 23:24:06.790381 8565 consensus_queue.cc:237] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.790856 8429 raft_consensus.cc:2454] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.792711 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.774630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38325 (local address 127.0.97.3:35865)
0926 23:24:06.784178 (+ 9548us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:06.784200 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:06.784307 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:06.785551 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:06.785556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:06.785594 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:06.785759 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.785769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.787607 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.787613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.789737 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:06.789752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.791761 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.791772 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:06.792032 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.792075 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:24:06.792204 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 23:24:06.792370 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9435,"thread_start_us":9884,"threads_started":1}
I20260926 23:24:06.793462 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.774993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35865 (local address 127.0.97.2:38325)
0926 23:24:06.785060 (+ 10067us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.785067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.785091 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:24:06.785224 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.785235 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.785347 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.785947 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:06.786127 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.787315 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.790051 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.791554 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.791697 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.791801 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.792969 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.793120 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9950,"thread_start_us":11217,"threads_started":1}
I20260926 23:24:06.794107 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e93d79e995b4fd3b8fda1be65a864ff" candidate_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" is_pre_election: true
I20260926 23:24:06.794448 8293 raft_consensus.cc:2379] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8fdcad86a04f4b0388c3537b07ccb3a1 in current term 1: Already voted for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in this term.
I20260926 23:24:06.794171 7991 catalog_manager.cc:5495] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 reported cstate change: term changed from 0 to 1, leader changed from <none> to f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } health_report { overall_health: UNKNOWN } } }
I20260926 23:24:06.794978 8360 leader_election.cc:302] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1; no voters: c553c6025f1a42e3a050f2e7ea31cde2, f583e7ac98ff4f2eb80d1d783ad3b338
I20260926 23:24:06.795449 8522 raft_consensus.cc:2726] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 23:24:06.831313 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:06.825504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.0.97.1:43483)
0926 23:24:06.825766 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:06.825772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:06.825793 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:24:06.825975 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:06.825981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:06.826086 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:06.826202 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:24:06.826546 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.827681 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.829152 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:06.829844 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:06.829978 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:06.830450 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:24:06.830629 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:24:06.830723 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:06.830902 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 23:24:06.830996 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260926 23:24:06.834138 8429 raft_consensus.cc:1261] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.834154 8293 raft_consensus.cc:1261] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.834779 8565 consensus_queue.cc:1035] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:06.835091 8521 consensus_queue.cc:1035] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:06.849344 7993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:06.849618 7993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:06.853511 7993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Sending DeleteTablet for 3 replicas of tablet 8e93d79e995b4fd3b8fda1be65a864ff
I20260926 23:24:06.854476 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57442
I20260926 23:24:06.854775 8532 tablet_replica.cc:330] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: stopping tablet replica
I20260926 23:24:06.854954 8532 raft_consensus.cc:2229] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:06.855186 8532 raft_consensus.cc:2258] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:06.855116 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38350
I20260926 23:24:06.855420 8270 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38350
I20260926 23:24:06.855489 8131 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35876
I20260926 23:24:06.855803 8130 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e93d79e995b4fd3b8fda1be65a864ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:06 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35876
I20260926 23:24:06.855839 8533 tablet_replica.cc:330] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: stopping tablet replica
I20260926 23:24:06.856061 8533 raft_consensus.cc:2229] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:24:06.856072 8534 tablet_replica.cc:330] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: stopping tablet replica
I20260926 23:24:06.856262 8534 raft_consensus.cc:2229] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:06.856468 8533 raft_consensus.cc:2258] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:06.856491 8534 raft_consensus.cc:2258] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:06.858161 8532 ts_tablet_manager.cc:1905] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:06.858335 8534 ts_tablet_manager.cc:1905] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:06.858861 8533 ts_tablet_manager.cc:1905] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:06.862648 7993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:24:06.863386 7993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:24:06.865109 8534 ts_tablet_manager.cc:1918] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:06.865262 8534 log.cc:1196] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/8e93d79e995b4fd3b8fda1be65a864ff
I20260926 23:24:06.865679 8534 ts_tablet_manager.cc:1939] T 8e93d79e995b4fd3b8fda1be65a864ff P c553c6025f1a42e3a050f2e7ea31cde2: Deleting consensus metadata
I20260926 23:24:06.865821 8532 ts_tablet_manager.cc:1918] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:06.866034 8532 log.cc:1196] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/8e93d79e995b4fd3b8fda1be65a864ff
I20260926 23:24:06.866582 8532 ts_tablet_manager.cc:1939] T 8e93d79e995b4fd3b8fda1be65a864ff P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting consensus metadata
I20260926 23:24:06.866835 7978 catalog_manager.cc:4841] TS c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325): tablet 8e93d79e995b4fd3b8fda1be65a864ff (table test-table [id=621c3f96d11048489bee21b97372a521]) successfully deleted
I20260926 23:24:06.868057 7981 catalog_manager.cc:4841] TS 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507): tablet 8e93d79e995b4fd3b8fda1be65a864ff (table test-table [id=621c3f96d11048489bee21b97372a521]) successfully deleted
W20260926 23:24:06.868049 7978 catalog_manager.cc:4808] TS c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325): delete failed for tablet 8e93d79e995b4fd3b8fda1be65a864ff because the tablet was not found. No further retry: Not found: Tablet not found: 8e93d79e995b4fd3b8fda1be65a864ff
I20260926 23:24:06.869853 8533 ts_tablet_manager.cc:1918] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:06.870018 8533 log.cc:1196] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/8e93d79e995b4fd3b8fda1be65a864ff
I20260926 23:24:06.870471 8533 ts_tablet_manager.cc:1939] T 8e93d79e995b4fd3b8fda1be65a864ff P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting consensus metadata
I20260926 23:24:06.871704 7978 catalog_manager.cc:4841] TS f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483): tablet 8e93d79e995b4fd3b8fda1be65a864ff (table test-table [id=621c3f96d11048489bee21b97372a521]) successfully deleted
W20260926 23:24:06.872123 7978 catalog_manager.cc:4808] TS f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483): delete failed for tablet 8e93d79e995b4fd3b8fda1be65a864ff because the tablet was not found. No further retry: Not found: Tablet not found: 8e93d79e995b4fd3b8fda1be65a864ff
I20260926 23:24:06.875102 8130 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 (DEFAULT_TABLE table=test-table [id=b345337c22c2433796215913ed422bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:06.875109 8270 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 (DEFAULT_TABLE table=test-table [id=b345337c22c2433796215913ed422bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:06.875554 8270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.875567 8130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.877393 8410 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 (DEFAULT_TABLE table=test-table [id=b345337c22c2433796215913ed422bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:06.877835 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:06.882299 8536 tablet_bootstrap.cc:492] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: Bootstrap starting.
I20260926 23:24:06.883936 8536 tablet_bootstrap.cc:654] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.886739 8536 tablet_bootstrap.cc:492] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: No bootstrap required, opened a new log
I20260926 23:24:06.886946 8536 ts_tablet_manager.cc:1397] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 23:24:06.887923 8536 raft_consensus.cc:348] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.888242 8536 raft_consensus.cc:374] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.888355 8536 raft_consensus.cc:729] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.888666 8536 consensus_queue.cc:260] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.889394 8536 ts_tablet_manager.cc:1428] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 23:24:06.891314 8535 tablet_bootstrap.cc:492] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: Bootstrap starting.
I20260926 23:24:06.891767 8537 tablet_bootstrap.cc:492] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: Bootstrap starting.
I20260926 23:24:06.893198 8535 tablet_bootstrap.cc:654] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.893463 8537 tablet_bootstrap.cc:654] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:06.896171 8537 tablet_bootstrap.cc:492] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: No bootstrap required, opened a new log
I20260926 23:24:06.896332 8537 ts_tablet_manager.cc:1397] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 23:24:06.897068 8537 raft_consensus.cc:348] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.897339 8537 raft_consensus.cc:374] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.897415 8537 raft_consensus.cc:729] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c553c6025f1a42e3a050f2e7ea31cde2, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.897665 8537 consensus_queue.cc:260] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.897837 8535 tablet_bootstrap.cc:492] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: No bootstrap required, opened a new log
I20260926 23:24:06.898043 8535 ts_tablet_manager.cc:1397] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 23:24:06.898171 8537 ts_tablet_manager.cc:1428] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:24:06.899441 8535 raft_consensus.cc:348] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.899695 8535 raft_consensus.cc:374] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:06.899794 8535 raft_consensus.cc:729] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdcad86a04f4b0388c3537b07ccb3a1, State: Initialized, Role: FOLLOWER
I20260926 23:24:06.900098 8535 consensus_queue.cc:260] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.901275 8535 ts_tablet_manager.cc:1428] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 23:24:06.911388 8565 raft_consensus.cc:482] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:06.911597 8565 raft_consensus.cc:504] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.912339 8565 leader_election.cc:288] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.912806 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7cf8ebbc544e6eb85b8d36e96d12b4" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" is_pre_election: true
I20260926 23:24:06.913050 8293 raft_consensus.cc:2454] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.913024 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7cf8ebbc544e6eb85b8d36e96d12b4" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" is_pre_election: true
I20260926 23:24:06.913278 8429 raft_consensus.cc:2454] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 0.
I20260926 23:24:06.913511 8081 leader_election.cc:302] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c553c6025f1a42e3a050f2e7ea31cde2, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.913796 8565 raft_consensus.cc:2781] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:24:06.913939 8565 raft_consensus.cc:482] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:24:06.914053 8565 raft_consensus.cc:3037] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.916162 8565 raft_consensus.cc:504] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.916770 8565 leader_election.cc:288] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Requested vote from peers c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325), 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507)
I20260926 23:24:06.917250 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7cf8ebbc544e6eb85b8d36e96d12b4" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1"
I20260926 23:24:06.917263 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7cf8ebbc544e6eb85b8d36e96d12b4" candidate_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c553c6025f1a42e3a050f2e7ea31cde2"
I20260926 23:24:06.917472 8429 raft_consensus.cc:3037] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.917472 8293 raft_consensus.cc:3037] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:06.920404 8429 raft_consensus.cc:2454] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.920404 8293 raft_consensus.cc:2454] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f583e7ac98ff4f2eb80d1d783ad3b338 in term 1.
I20260926 23:24:06.920856 8084 leader_election.cc:302] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fdcad86a04f4b0388c3537b07ccb3a1, f583e7ac98ff4f2eb80d1d783ad3b338; no voters:
I20260926 23:24:06.921190 8565 raft_consensus.cc:2781] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:06.921363 8565 raft_consensus.cc:686] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 LEADER]: Becoming Leader. State: Replica: f583e7ac98ff4f2eb80d1d783ad3b338, State: Running, Role: LEADER
I20260926 23:24:06.921653 8565 consensus_queue.cc:237] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } }
I20260926 23:24:06.924338 7993 catalog_manager.cc:5495] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 reported cstate change: term changed from 0 to 1, leader changed from <none> to f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f583e7ac98ff4f2eb80d1d783ad3b338" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 } health_report { overall_health: UNKNOWN } } }
I20260926 23:24:06.954165 8293 raft_consensus.cc:1261] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.954268 8429 raft_consensus.cc:1261] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Refusing update from remote peer f583e7ac98ff4f2eb80d1d783ad3b338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:06.954761 8565 consensus_queue.cc:1035] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fdcad86a04f4b0388c3537b07ccb3a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:06.955027 8521 consensus_queue.cc:1035] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c553c6025f1a42e3a050f2e7ea31cde2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:07.033864 7994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:07.034148 7994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:07.039047 7994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66518814d1724995add55f1119f9db43: Sending DeleteTablet for 3 replicas of tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4
I20260926 23:24:07.040036 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57442
I20260926 23:24:07.040097 8270 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38350
I20260926 23:24:07.040376 8534 tablet_replica.cc:330] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: stopping tablet replica
I20260926 23:24:07.040400 8532 tablet_replica.cc:330] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: stopping tablet replica
I20260926 23:24:07.040606 8532 raft_consensus.cc:2229] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:07.040634 8534 raft_consensus.cc:2229] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:07.040832 8532 raft_consensus.cc:2258] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:07.040874 8534 raft_consensus.cc:2258] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:07.042006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 8065
I20260926 23:24:07.042014 8532 ts_tablet_manager.cc:1905] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:07.042462 8534 ts_tablet_manager.cc:1905] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:07.045222 8130 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35876
I20260926 23:24:07.045524 8533 tablet_replica.cc:330] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: stopping tablet replica
I20260926 23:24:07.045756 8533 raft_consensus.cc:2229] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:24:07.046142 8533 raft_consensus.cc:2258] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:07.047233 8533 ts_tablet_manager.cc:1905] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:07.052978 8534 ts_tablet_manager.cc:1918] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:24:07.053423 8534 log.cc:1196] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/1c7cf8ebbc544e6eb85b8d36e96d12b4
I20260926 23:24:07.053973 8534 ts_tablet_manager.cc:1939] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P c553c6025f1a42e3a050f2e7ea31cde2: Deleting consensus metadata
I20260926 23:24:07.054368 8533 ts_tablet_manager.cc:1918] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:24:07.054555 8533 log.cc:1196] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/1c7cf8ebbc544e6eb85b8d36e96d12b4
I20260926 23:24:07.054989 8533 ts_tablet_manager.cc:1939] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P f583e7ac98ff4f2eb80d1d783ad3b338: Deleting consensus metadata
I20260926 23:24:07.055624 7978 catalog_manager.cc:4841] TS c553c6025f1a42e3a050f2e7ea31cde2 (127.0.97.2:38325): tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 (table test-table [id=b345337c22c2433796215913ed422bb9]) successfully deleted
I20260926 23:24:07.056113 7978 catalog_manager.cc:4841] TS f583e7ac98ff4f2eb80d1d783ad3b338 (127.0.97.1:43483): tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 (table test-table [id=b345337c22c2433796215913ed422bb9]) successfully deleted
I20260926 23:24:07.060009 8532 ts_tablet_manager.cc:1918] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:24:07.060191 8532 log.cc:1196] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/1c7cf8ebbc544e6eb85b8d36e96d12b4
I20260926 23:24:07.060591 8532 ts_tablet_manager.cc:1939] T 1c7cf8ebbc544e6eb85b8d36e96d12b4 P 8fdcad86a04f4b0388c3537b07ccb3a1: Deleting consensus metadata
I20260926 23:24:07.061842 7981 catalog_manager.cc:4841] TS 8fdcad86a04f4b0388c3537b07ccb3a1 (127.0.97.3:33507): tablet 1c7cf8ebbc544e6eb85b8d36e96d12b4 (table test-table [id=b345337c22c2433796215913ed422bb9]) successfully deleted
Sep 26 23:24:07 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 23:24:07.071491 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:07.044765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.0.97.1:43483)
0926 23:24:07.045036 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:07.045066 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:07.053545 (+ 8479us) server_negotiation.cc:407] Connection header received
0926 23:24:07.054065 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:07.054077 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:07.054196 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:07.054292 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:07.056422 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:07.057459 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:07.058870 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:07.059371 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:07.059540 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:07.064504 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:07.064520 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:07.064538 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:07.064582 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:07.068302 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:07.069195 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:07.069204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:07.069207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:07.069357 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:07.069798 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:07.069802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:07.069804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:07.070216 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:07.070568 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:07.070883 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 23:24:07.071146 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:24:07.072229 8196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:07.232106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 8065
I20260926 23:24:07.257001 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 8205
Sep 26 23:24:07 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 23:24:07.271358 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:07.257704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.0.97.2:38325)
0926 23:24:07.257921 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:07.257928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:07.258024 (+ 96us) server_negotiation.cc:407] Connection header received
0926 23:24:07.258246 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:07.258251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:07.258327 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:07.258447 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:07.259603 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:07.260306 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:07.261698 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:07.262109 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:07.262243 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:07.266368 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:07.266381 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:07.266396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:07.266436 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:07.269119 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:07.269723 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:07.269730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:07.269732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:07.269856 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:07.270161 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:07.270164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:07.270166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:07.270516 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:07.270739 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:07.270947 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 23:24:07.271103 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:24:07.272089 8335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:07.393631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 8205
I20260926 23:24:07.414836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 8344
Sep 26 23:24:07 dist-test-slave-q5g8 krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 23:24:07.429477 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:07.415530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.0.97.3:33507)
0926 23:24:07.415967 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:07.415972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:07.415987 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:24:07.416039 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:07.416043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:07.416111 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:07.416199 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:07.417631 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:07.418304 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:07.419458 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:07.419856 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:07.419953 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:07.424177 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:07.424190 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:07.424207 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:07.424242 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:07.427153 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:07.427706 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:07.427713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:07.427715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:07.427833 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:07.428131 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:07.428133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:07.428135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:07.428457 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:07.428771 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:07.428935 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 23:24:07.429100 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20260926 23:24:07.430166 8475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:07.526476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 8344
I20260926 23:24:07.544049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 7962
I20260926 23:24:07.545204 8034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:07.658221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 7962
2026-09-26T23:24:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5865 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:24:07 dist-test-slave-q5g8 krb5kdc[8608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:24:07 dist-test-slave-q5g8 krb5kdc[8608](info): set up 2 sockets
Sep 26 23:24:07 dist-test-slave-q5g8 krb5kdc[8608](info): commencing operation
krb5kdc: starting...
W20260926 23:24:08.758102 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:24:08 dist-test-slave-q5g8 krb5kdc[8608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:24:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:24:08.968556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39211
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:09.284844 8621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:09.285121 8621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:09.285211 8621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:09.292280 8621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:24:09.292387 8621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:09.292439 8621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:09.292495 8621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:24:09.292542 8621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:24:09.302330 8621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:09.303545 8621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:09.304777 8621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:09.316246 8627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:09.316577 8626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:09.316469 8629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:09.317725 8621 server_base.cc:1034] running on GCE node
I20260926 23:24:09.318457 8621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:09.319769 8621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:09.320964 8621 hybrid_clock.cc:648] HybridClock initialized: now 1790465049320929 us; error 34 us; skew 500 ppm
I20260926 23:24:09.321305 8621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T23:24:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1735 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:24:09 dist-test-slave-q5g8 krb5kdc[8635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:24:09 dist-test-slave-q5g8 krb5kdc[8635](info): set up 2 sockets
Sep 26 23:24:09 dist-test-slave-q5g8 krb5kdc[8635](info): commencing operation
krb5kdc: starting...
W20260926 23:24:10.513646 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:24:10 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:24:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:24:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:24:10.736471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44579
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:11.056716 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:11.057005 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:11.057194 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:11.064651 8648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:24:11.064821 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:11.064922 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:11.065012 8648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:24:11.065086 8648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:24:11.075806 8648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:11.077220 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:11.078578 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:11.090430 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:11.090505 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:11.090479 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:11.096905 8648 server_base.cc:1034] running on GCE node
I20260926 23:24:11.097762 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:11.099265 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:11.100505 8648 hybrid_clock.cc:648] HybridClock initialized: now 1790465051100448 us; error 63 us; skew 500 ppm
I20260926 23:24:11.100929 8648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:11 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:11.104056 8648 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:24:11.105695 8648 webserver.cc:460] Webserver started at http://127.0.97.62:40197/ using document root <none> and password file <none>
I20260926 23:24:11.106436 8648 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:11.106532 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:11.106805 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:11.109861 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "1d1083d737c14df29a237a6ba77d28ae"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "6b630a3540eec84ef61f0fbabb7845fe"
server_key_iv: "9d2abff46f4879f505f0ef049f117ecb"
server_key_version: "encryptionkey@0"
I20260926 23:24:11.110618 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "1d1083d737c14df29a237a6ba77d28ae"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "6b630a3540eec84ef61f0fbabb7845fe"
server_key_iv: "9d2abff46f4879f505f0ef049f117ecb"
server_key_version: "encryptionkey@0"
I20260926 23:24:11.116396 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:24:11.120033 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:11.121537 8648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:11.121711 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "1d1083d737c14df29a237a6ba77d28ae"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "6b630a3540eec84ef61f0fbabb7845fe"
server_key_iv: "9d2abff46f4879f505f0ef049f117ecb"
server_key_version: "encryptionkey@0"
I20260926 23:24:11.121899 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:11.154291 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:11.159003 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:11.159361 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:11.179131 8648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44579
I20260926 23:24:11.179162 8714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44579 every 8 connection(s)
I20260926 23:24:11.180605 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:24:11.184650 8715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:11.187705 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 8648
I20260926 23:24:11.187937 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:24:11.188800 388 external_mini_cluster.cc:1428] Setting key 4149201f6ac4e264dc35259091526fd4
I20260926 23:24:11.197769 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Bootstrap starting.
I20260926 23:24:11.201529 8715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:11.202456 8715 log.cc:824] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:11.206071 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: No bootstrap required, opened a new log
Sep 26 23:24:11 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:11.215104 8715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:11.215593 8715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:11.215731 8715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1083d737c14df29a237a6ba77d28ae, State: Initialized, Role: FOLLOWER
I20260926 23:24:11.215988 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:11.190951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.0.97.62:44579)
0926 23:24:11.191739 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:11.191752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:11.191793 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:24:11.193539 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:11.193562 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:11.193989 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:11.194330 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:11.196141 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:11.197513 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:11.199235 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:11.199912 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:11.200044 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:11.208319 (+ 8275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:11.208328 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:11.208344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:11.208388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:11.211578 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:11.212304 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:11.212311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:11.212318 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:11.212480 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:11.212941 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:11.212947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:11.212950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:11.213526 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:11.213838 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:11.214278 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 23:24:11.214715 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":373,"threads_started":1}
I20260926 23:24:11.216425 8715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:11.216780 8715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:24:11.216948 8715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:24:11.217133 8715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:11.220760 8715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:11.221300 8715 leader_election.cc:302] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d1083d737c14df29a237a6ba77d28ae; no voters:
I20260926 23:24:11.221939 8715 leader_election.cc:288] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:24:11.222082 8720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:11.222786 8720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 LEADER]: Becoming Leader. State: Replica: 1d1083d737c14df29a237a6ba77d28ae, State: Running, Role: LEADER
I20260926 23:24:11.223321 8720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:11.224107 8715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:24:11.226276 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d1083d737c14df29a237a6ba77d28ae. Latest consensus state: current_term: 1 leader_uuid: "1d1083d737c14df29a237a6ba77d28ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } } }
I20260926 23:24:11.226675 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:11.226564 8721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d1083d737c14df29a237a6ba77d28ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } } }
I20260926 23:24:11.226866 8721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:11.227913 8726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:24:11.235229 8726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:24:11.245194 8726 catalog_manager.cc:1347] Generated new cluster ID: f0495a074b93402eafacfbb8e161fe62
I20260926 23:24:11.245363 8726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:24:11.264091 8726 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:24:11.265579 8726 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:24:11.272508 8726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Generated new TSK 0
I20260926 23:24:11.273379 8726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:24:11.329200 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:11.692940 8741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:11.693238 8741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:11.693446 8741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:11.700659 8741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:11.700845 8741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:11.701035 8741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:24:11.711151 8741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:11.712759 8741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:11.714215 8741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:11.728466 8746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:11.728539 8747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:11.728499 8749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:11.729274 8741 server_base.cc:1034] running on GCE node
I20260926 23:24:11.730023 8741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:11.730986 8741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:11.732275 8741 hybrid_clock.cc:648] HybridClock initialized: now 1790465051732146 us; error 29 us; skew 500 ppm
I20260926 23:24:11.732746 8741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:11 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:11.736244 8741 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:24:11.737975 8741 webserver.cc:460] Webserver started at http://127.0.97.1:44733/ using document root <none> and password file <none>
I20260926 23:24:11.738708 8741 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:11.738842 8741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:11.739171 8741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:11.742218 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "d8555db74fdd4d30865b9331de9b4b72"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "861a6a3055aeec727e564e8a86d152dc"
server_key_iv: "cfee78355fac49f140be34a0a20e6307"
server_key_version: "encryptionkey@0"
I20260926 23:24:11.743000 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d8555db74fdd4d30865b9331de9b4b72"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "861a6a3055aeec727e564e8a86d152dc"
server_key_iv: "cfee78355fac49f140be34a0a20e6307"
server_key_version: "encryptionkey@0"
I20260926 23:24:11.748284 8741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:24:11.751888 8756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:11.753301 8741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:11.753485 8741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "d8555db74fdd4d30865b9331de9b4b72"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "861a6a3055aeec727e564e8a86d152dc"
server_key_iv: "cfee78355fac49f140be34a0a20e6307"
server_key_version: "encryptionkey@0"
I20260926 23:24:11.753687 8741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:11.780901 8741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:11.785183 8741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:11.785485 8741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:11.786445 8741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:11.787833 8741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:11.787940 8741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:11.788084 8741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:11.788156 8741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:11.821071 8741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42621
I20260926 23:24:11.821120 8868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42621 every 8 connection(s)
I20260926 23:24:11.822374 8741 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:24:11.825286 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 8741
I20260926 23:24:11.825505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:24:11.826392 388 external_mini_cluster.cc:1428] Setting key ac30401a7f84c658547c64a0acfb78f6
Sep 26 23:24:11 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:11.842629 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:11.824852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53477 (local address 127.0.97.62:44579)
0926 23:24:11.825306 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:11.825311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:11.826982 (+ 1671us) server_negotiation.cc:407] Connection header received
0926 23:24:11.828887 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:11.828891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:11.828971 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:11.829092 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:11.830579 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:11.831287 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:11.832878 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:11.833289 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:11.833433 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:11.837144 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:11.837149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:11.837152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:11.837182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:11.839600 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:11.840336 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:11.840340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:11.840342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:11.840447 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:11.840934 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:11.840939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:11.840943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:11.841194 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:11.841362 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:11.842049 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 23:24:11.842259 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":260,"threads_started":1}
I20260926 23:24:11.843600 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:11.825511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:53477)
0926 23:24:11.826374 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:11.826416 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:11.828565 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:11.829331 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:11.829339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:11.829871 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:11.830422 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:11.830438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:11.831455 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:11.831465 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:11.832729 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:11.832753 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:11.833526 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:11.833534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:11.833787 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:11.834482 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:11.834503 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:11.836928 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:11.839744 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:11.839753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:11.839791 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:11.840192 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:11.840557 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:11.840563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:11.840567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:11.840820 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:11.841336 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:11.841344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:11.841679 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 23:24:11.841942 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 23:24:11.842459 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":346,"threads_started":1}
I20260926 23:24:11.845250 8869 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44579
I20260926 23:24:11.845595 8869 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:11.846515 8869 heartbeater.cc:511] Master 127.0.97.62:44579 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:24:11.848973 8680 ts_manager.cc:194] Registered new tserver with Master: d8555db74fdd4d30865b9331de9b4b72 (127.0.97.1:42621)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:24:11.850472 8680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:24:11.876153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:12.205448 8875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:12.205734 8875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:12.205892 8875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:12.213306 8875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:12.213434 8875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:12.213613 8875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:24:12.223688 8875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:12.225211 8875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:12.226629 8875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:12.240622 8880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:12.240625 8881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:12.240787 8883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:12.241375 8875 server_base.cc:1034] running on GCE node
I20260926 23:24:12.242076 8875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:12.243052 8875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:12.244277 8875 hybrid_clock.cc:648] HybridClock initialized: now 1790465052244204 us; error 67 us; skew 500 ppm
I20260926 23:24:12.244659 8875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:12 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:12.248190 8875 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:24:12.249840 8875 webserver.cc:460] Webserver started at http://127.0.97.2:36225/ using document root <none> and password file <none>
I20260926 23:24:12.250563 8875 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:12.250701 8875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:12.251022 8875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:12.254078 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "c7cce50880b94ddfb14156fa04b783d1"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "89f8d968858bc3aba1b65b1e6d45d208"
server_key_iv: "535231fa6a1c00135ef9b867787d34d3"
server_key_version: "encryptionkey@0"
I20260926 23:24:12.254932 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c7cce50880b94ddfb14156fa04b783d1"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "89f8d968858bc3aba1b65b1e6d45d208"
server_key_iv: "535231fa6a1c00135ef9b867787d34d3"
server_key_version: "encryptionkey@0"
I20260926 23:24:12.260725 8875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:24:12.264387 8890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:12.265967 8875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:12.266165 8875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "c7cce50880b94ddfb14156fa04b783d1"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "89f8d968858bc3aba1b65b1e6d45d208"
server_key_iv: "535231fa6a1c00135ef9b867787d34d3"
server_key_version: "encryptionkey@0"
I20260926 23:24:12.266371 8875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:12.282620 8875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:12.286751 8875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:12.287045 8875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:12.288030 8875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:12.289388 8875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:12.289490 8875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:12.289644 8875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:12.289717 8875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:12.327209 8875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44761
I20260926 23:24:12.327301 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44761 every 8 connection(s)
I20260926 23:24:12.328632 8875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:24:12.338662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 8875
I20260926 23:24:12.338874 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:24:12.339411 388 external_mini_cluster.cc:1428] Setting key a3d2f342afa1e9818b9c7134476ff822
Sep 26 23:24:12 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:12.349690 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:12.331286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47605 (local address 127.0.97.62:44579)
0926 23:24:12.331455 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:12.331460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:12.332712 (+ 1252us) server_negotiation.cc:407] Connection header received
0926 23:24:12.334477 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:12.334481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:12.334545 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:12.334634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:12.336020 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.336812 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.338305 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.338798 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.338900 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:12.342876 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:12.342885 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:12.342893 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:12.342939 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:12.346474 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:12.347225 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:12.347233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:12.347240 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:12.347388 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:12.347819 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:12.347826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:12.347832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:12.348199 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:12.348347 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:12.349145 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 23:24:12.349380 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:24:12.351198 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:12.331834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:47605)
0926 23:24:12.332503 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:12.332546 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:12.334227 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:12.334830 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:12.334838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:12.335337 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:12.335868 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:12.335884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.336935 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.336939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:12.338170 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:12.338181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.339023 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.339033 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:12.339343 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:12.340066 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:12.340093 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:12.342618 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:12.346644 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:12.346651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:12.346666 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:12.347096 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:12.347501 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:12.347504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:12.347507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:12.347705 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:12.348318 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:12.348323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:12.348658 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 23:24:12.349080 (+ 422us) client_negotiation.cc:240] Negotiation successful
0926 23:24:12.349686 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260926 23:24:12.352780 9003 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44579
I20260926 23:24:12.353086 9003 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:12.353796 9003 heartbeater.cc:511] Master 127.0.97.62:44579 requested a full tablet report, sending...
I20260926 23:24:12.355356 8680 ts_manager.cc:194] Registered new tserver with Master: c7cce50880b94ddfb14156fa04b783d1 (127.0.97.2:44761)
I20260926 23:24:12.356609 8680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47605
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:24:12.392577 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:12.716010 9008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:12.716363 9008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:12.716538 9008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:12.723747 9008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:12.723889 9008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:12.724067 9008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:24:12.734076 9008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:12.735488 9008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:12.736894 9008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:12.749810 9014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:12.749879 9013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:12.749902 9016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:12.751031 9008 server_base.cc:1034] running on GCE node
I20260926 23:24:12.751675 9008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:12.752588 9008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:12.753815 9008 hybrid_clock.cc:648] HybridClock initialized: now 1790465052753761 us; error 48 us; skew 500 ppm
I20260926 23:24:12.754129 9008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:12 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:12.757277 9008 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:24:12.758678 9008 webserver.cc:460] Webserver started at http://127.0.97.3:36995/ using document root <none> and password file <none>
I20260926 23:24:12.759343 9008 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:12.759479 9008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:12.759779 9008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:12.762704 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "8676a2c9b83c419d80a08899018127e4"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "2e9b635d8abe92bd44572e01b86701a3"
server_key_iv: "bfc307d0add28dacca44abaa986e4843"
server_key_version: "encryptionkey@0"
I20260926 23:24:12.763464 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8676a2c9b83c419d80a08899018127e4"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "2e9b635d8abe92bd44572e01b86701a3"
server_key_iv: "bfc307d0add28dacca44abaa986e4843"
server_key_version: "encryptionkey@0"
I20260926 23:24:12.768500 9008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 23:24:12.772084 9023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:12.773525 9008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:12.773676 9008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "8676a2c9b83c419d80a08899018127e4"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "2e9b635d8abe92bd44572e01b86701a3"
server_key_iv: "bfc307d0add28dacca44abaa986e4843"
server_key_version: "encryptionkey@0"
I20260926 23:24:12.773850 9008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:12.800221 9008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:12.804440 9008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:12.804688 9008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:12.805709 9008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:12.806945 9008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:12.807016 9008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:12.807096 9008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:12.807138 9008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:12.841274 9008 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36723
I20260926 23:24:12.841318 9135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36723 every 8 connection(s)
I20260926 23:24:12.842566 9008 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:24:12.843259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9008
I20260926 23:24:12.843478 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:24:12.844013 388 external_mini_cluster.cc:1428] Setting key 04b14977a094b8976e7d042b924d2b89
I20260926 23:24:12.853354 8869 heartbeater.cc:503] Master 127.0.97.62:44579 was elected leader, sending a full tablet report...
Sep 26 23:24:12 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:12.863260 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:12.845219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43783 (local address 127.0.97.62:44579)
0926 23:24:12.845400 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:12.845404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:12.846756 (+ 1352us) server_negotiation.cc:407] Connection header received
0926 23:24:12.848754 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:12.848759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:12.848843 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:12.848958 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:12.850462 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.851373 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.853007 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.853410 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.853560 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:12.857203 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:12.857208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:12.857213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:12.857243 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:12.860419 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:12.861204 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:12.861208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:12.861211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:12.861341 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:12.861732 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:12.861735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:12.861737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:12.861995 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:12.862171 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:12.862790 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 23:24:12.862968 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 23:24:12.864200 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:12.845839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:43783)
0926 23:24:12.846603 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:12.846645 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:12.848458 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:12.849208 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:12.849216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:12.849702 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:12.850263 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:12.850280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.851523 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.851528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:12.852857 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:12.852873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:12.853666 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:12.853677 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:12.853896 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:12.854638 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:12.854666 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:12.856971 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:12.860544 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:12.860552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:12.860567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:12.861078 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:12.861452 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:12.861455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:12.861457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:12.861630 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:12.862106 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:12.862113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:12.862432 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 23:24:12.862650 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 23:24:12.863065 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":336,"threads_started":1}
I20260926 23:24:12.865609 9136 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44579
I20260926 23:24:12.865921 9136 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:12.866618 9136 heartbeater.cc:511] Master 127.0.97.62:44579 requested a full tablet report, sending...
I20260926 23:24:12.867942 8680 ts_manager.cc:194] Registered new tserver with Master: 8676a2c9b83c419d80a08899018127e4 (127.0.97.3:36723)
I20260926 23:24:12.868803 8680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43783
I20260926 23:24:12.872275 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 23:24:13.015470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 8741
Sep 26 23:24:13 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:24:13.032804 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:13.016802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.97.1:42621)
0926 23:24:13.017616 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:13.017624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:13.017652 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:24:13.017712 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:13.017719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:13.017971 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:13.018107 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:13.019429 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:13.020504 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:13.021763 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:13.022347 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:13.022446 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:13.026892 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:13.026907 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:13.026922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:13.026963 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:13.030073 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:13.030809 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:13.030815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:13.030818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:13.030949 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:13.031288 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:13.031292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:13.031294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:13.031651 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:13.031871 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:13.032143 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:24:13.032343 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":473,"threads_started":1}
I20260926 23:24:13.034066 8864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:13.118258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 8741
I20260926 23:24:13.134632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 8875
Sep 26 23:24:13 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:24:13.151049 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:13.135454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.0.97.2:44761)
0926 23:24:13.136060 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:13.136068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:13.136093 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:24:13.136160 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:13.136166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:13.136379 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:13.136517 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:13.137785 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:13.139022 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:13.140271 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:13.141085 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:13.141212 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:13.145486 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:13.145499 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:13.145514 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:13.145559 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:13.148518 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:13.149276 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:13.149284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:13.149286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:13.149424 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:13.149739 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:13.149743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:13.149745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:13.150119 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:13.150353 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:13.150566 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 23:24:13.150757 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":349,"threads_started":1}
I20260926 23:24:13.152376 8998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:13.235001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 8875
I20260926 23:24:13.252369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9008
Sep 26 23:24:13 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:24:13.269830 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:13.253420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.97.3:36723)
0926 23:24:13.254106 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:13.254114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:13.254143 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:24:13.254223 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:13.254229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:13.254450 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:13.254575 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:13.255884 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:13.257092 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:13.258539 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:13.259102 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:13.259212 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:13.263869 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:13.263881 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:13.263898 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:13.263933 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:13.266947 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:13.267645 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:13.267652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:13.267654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:13.267780 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:13.268185 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:13.268188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:13.268190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:13.268516 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:13.268825 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:13.269121 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 23:24:13.269454 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":378,"threads_started":1}
I20260926 23:24:13.271186 9131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:13.354437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9008
I20260926 23:24:13.370450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 8648
I20260926 23:24:13.371683 8710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:13.473448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 8648
I20260926 23:24:13.493448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44579
--webserver_interface=127.0.97.62
--webserver_port=40197
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:13.821765 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:13.822108 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:13.822276 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:13.829265 9153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:24:13.829401 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:13.829510 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:13.829599 9153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:24:13.829665 9153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:24:13.839110 9153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:13.840415 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:13.841727 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:13.853704 9158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:13.853648 9161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:13.853727 9159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:13.855052 9153 server_base.cc:1034] running on GCE node
I20260926 23:24:13.855809 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:13.857270 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:13.858517 9153 hybrid_clock.cc:648] HybridClock initialized: now 1790465053858472 us; error 42 us; skew 500 ppm
I20260926 23:24:13.858857 9153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:13 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:13.861951 9153 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:24:13.863480 9153 webserver.cc:460] Webserver started at http://127.0.97.62:40197/ using document root <none> and password file <none>
I20260926 23:24:13.864152 9153 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:13.864301 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:13.869619 9153 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:13.872438 9168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:13.873806 9153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:24:13.873981 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "1d1083d737c14df29a237a6ba77d28ae"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "6b630a3540eec84ef61f0fbabb7845fe"
server_key_iv: "9d2abff46f4879f505f0ef049f117ecb"
server_key_version: "encryptionkey@0"
I20260926 23:24:13.874894 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:13.898722 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:13.902815 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:13.903081 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:13.921327 9153 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44579
I20260926 23:24:13.921368 9219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44579 every 8 connection(s)
I20260926 23:24:13.922758 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:24:13.924816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9153
I20260926 23:24:13.925726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42621
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44579
--webserver_port=44733
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:24:13.932238 9220 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:24:13.935925 9220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Bootstrap starting.
I20260926 23:24:13.954999 9220 log.cc:824] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:13.963829 9220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:24:13.964356 9220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Bootstrap complete.
I20260926 23:24:13.973372 9220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:13.974190 9220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1083d737c14df29a237a6ba77d28ae, State: Initialized, Role: FOLLOWER
I20260926 23:24:13.974896 9220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:13.975134 9220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:24:13.975247 9220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:24:13.975426 9220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:24:13.979537 9220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:13.980026 9220 leader_election.cc:302] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d1083d737c14df29a237a6ba77d28ae; no voters:
I20260926 23:24:13.980720 9220 leader_election.cc:288] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:24:13.980854 9224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:24:13.981958 9224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [term 2 LEADER]: Becoming Leader. State: Replica: 1d1083d737c14df29a237a6ba77d28ae, State: Running, Role: LEADER
I20260926 23:24:13.982676 9224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260926 23:24:13.982961 9220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:24:13.985785 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1d1083d737c14df29a237a6ba77d28ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } } }
I20260926 23:24:13.985584 9226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d1083d737c14df29a237a6ba77d28ae. Latest consensus state: current_term: 2 leader_uuid: "1d1083d737c14df29a237a6ba77d28ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1083d737c14df29a237a6ba77d28ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } } }
I20260926 23:24:13.986752 9226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:13.986989 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:13.994072 9232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:24:14.001034 9232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:24:14.002980 9232 catalog_manager.cc:1259] Loaded cluster ID: f0495a074b93402eafacfbb8e161fe62
I20260926 23:24:14.003131 9232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:24:14.007965 9232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:24:14.009895 9232 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d1083d737c14df29a237a6ba77d28ae: Loaded TSK: 0
I20260926 23:24:14.011844 9232 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:24:14.282572 9222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:14.282900 9222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:14.283068 9222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:14.290454 9222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:14.290598 9222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:14.290793 9222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:24:14.300257 9222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44733
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:14.301615 9222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:14.302956 9222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:14.315625 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:14.315773 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:14.315917 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:14.316545 9222 server_base.cc:1034] running on GCE node
I20260926 23:24:14.317278 9222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:14.318572 9222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:14.319800 9222 hybrid_clock.cc:648] HybridClock initialized: now 1790465054319713 us; error 56 us; skew 500 ppm
I20260926 23:24:14.320160 9222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465054, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:14.325522 9222 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:24:14.327744 9222 webserver.cc:460] Webserver started at http://127.0.97.1:44733/ using document root <none> and password file <none>
I20260926 23:24:14.328611 9222 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:14.328783 9222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:14.335564 9222 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:24:14.338644 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:14.340129 9222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:24:14.340279 9222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "d8555db74fdd4d30865b9331de9b4b72"
format_stamp: "Formatted at 2026-09-26 23:24:11 on dist-test-slave-q5g8"
server_key: "861a6a3055aeec727e564e8a86d152dc"
server_key_iv: "cfee78355fac49f140be34a0a20e6307"
server_key_version: "encryptionkey@0"
I20260926 23:24:14.341575 9222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:14.363499 9222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:14.368989 9222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:14.369273 9222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:14.370549 9222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:14.372267 9222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:14.372391 9222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:14.372501 9222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:14.372574 9222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:14.412599 9222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42621
I20260926 23:24:14.412654 9368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42621 every 8 connection(s)
I20260926 23:24:14.414049 9222 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:24:14.420516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9222
I20260926 23:24:14.421226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44761
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44579
--webserver_port=36225
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:14.433665 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:14.417194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:42295)
0926 23:24:14.417993 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:14.418042 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:14.419688 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:14.421296 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:14.421309 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:14.422055 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:14.422877 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.422903 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.424549 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.424557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.426697 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.426713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.427933 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.427938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.428140 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.428893 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:14.428915 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:14.432670 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":339,"threads_started":1}
W20260926 23:24:14.434286 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:14.417172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42295 (local address 127.0.97.62:44579)
0926 23:24:14.418321 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:14.418333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:14.418377 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:24:14.420311 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:14.420326 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:14.420736 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:14.421078 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:14.423092 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.424391 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.426868 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.427717 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.427920 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.433148 (+ 5228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42295: BlockingRecv error: recv got EOF from 127.0.97.1:42295 (error 108)
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":329,"threads_started":1}
W20260926 23:24:14.434352 9369 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44579 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44579: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:14.444494 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:14.435022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:52525)
0926 23:24:14.435235 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:14.435256 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:14.435358 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:14.435692 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:14.435699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:14.436149 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:14.436354 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.436367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.437455 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.437462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.439231 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.439249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.439917 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.439925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.440158 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.440808 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:14.440839 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:14.444142 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 23:24:14.444636 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:14.435277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52525 (local address 127.0.97.62:44579)
0926 23:24:14.435420 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:14.435424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:14.435439 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:24:14.435481 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:14.435485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:14.435539 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:14.435623 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:14.436490 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.437282 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.439381 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.439763 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.439855 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.444403 (+ 4548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52525: BlockingRecv error: recv got EOF from 127.0.97.1:52525 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:14.453145 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:14.445177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:49647)
0926 23:24:14.445414 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:14.445431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:14.445534 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:14.445855 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:14.445860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:14.446157 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:14.446314 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.446324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.447358 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.447364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.448601 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.448620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.449320 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.449326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.449505 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.450085 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:14.450101 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:14.452838 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260926 23:24:14.453223 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:14.445311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49647 (local address 127.0.97.62:44579)
0926 23:24:14.445441 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:14.445448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:14.445471 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:24:14.445652 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:14.445657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:14.445733 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:14.445833 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:14.446463 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.447227 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.448767 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.449181 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.449310 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.453011 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49647: BlockingRecv error: recv got EOF from 127.0.97.1:49647 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 23:24:14.453414 9369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:24:14.744516 9373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:14.744797 9373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:14.744985 9373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:14.752118 9373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:14.752220 9373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:14.752400 9373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:24:14.762156 9373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36225
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:14.763398 9373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:14.764727 9373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:14.777648 9381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:14.777716 9379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:14.778000 9378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:14.779002 9373 server_base.cc:1034] running on GCE node
I20260926 23:24:14.779650 9373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:14.780557 9373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:14.781776 9373 hybrid_clock.cc:648] HybridClock initialized: now 1790465054781752 us; error 61 us; skew 500 ppm
I20260926 23:24:14.782086 9373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465054, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:14.785111 9373 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:24:14.786542 9373 webserver.cc:460] Webserver started at http://127.0.97.2:36225/ using document root <none> and password file <none>
I20260926 23:24:14.787205 9373 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:14.787355 9373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:14.792416 9373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:24:14.795045 9388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:14.796235 9373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:24:14.796398 9373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "c7cce50880b94ddfb14156fa04b783d1"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "89f8d968858bc3aba1b65b1e6d45d208"
server_key_iv: "535231fa6a1c00135ef9b867787d34d3"
server_key_version: "encryptionkey@0"
I20260926 23:24:14.797385 9373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:14.816033 9373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:14.820389 9373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:14.820715 9373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:14.821681 9373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:14.822981 9373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:14.823117 9373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:14.823256 9373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:14.823347 9373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:14.856068 9373 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44761
I20260926 23:24:14.856159 9500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44761 every 8 connection(s)
I20260926 23:24:14.857554 9373 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:24:14.865271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9373
I20260926 23:24:14.865844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36723
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44579
--webserver_port=36995
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:24:14.872622 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:14.859981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35805 (local address 127.0.97.62:44579)
0926 23:24:14.860147 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:14.860151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:14.861284 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 23:24:14.863053 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:14.863058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:14.863142 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:14.863248 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:14.864712 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.865476 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.867035 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.867432 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.867562 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.872318 (+ 4756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35805: BlockingRecv error: recv got EOF from 127.0.97.2:35805 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:24:14.873134 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:14.860399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:35805)
0926 23:24:14.861130 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:14.861171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:14.862789 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:14.863471 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:14.863478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:14.863952 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:14.864479 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.864495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.865620 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.865627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.866888 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.866900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.867662 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.867669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.867883 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.868540 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:14.868566 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:14.872062 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":316,"threads_started":1}
W20260926 23:24:14.873908 9501 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44579 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44579: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:14.883287 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:14.874465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:34649)
0926 23:24:14.874667 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:14.874688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:14.874788 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:14.875083 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:14.875090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:14.875510 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:14.875719 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.875733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.876643 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.876651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.878375 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.878391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.879190 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.879199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.879410 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.879965 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:14.879992 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:14.883046 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:24:14.883991 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:14.874614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34649 (local address 127.0.97.62:44579)
0926 23:24:14.874745 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:14.874750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:14.874766 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:24:14.874889 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:14.874892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:14.874948 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:14.875028 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:14.875849 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.876512 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.878531 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.879037 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.879138 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.883357 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34649: BlockingRecv error: recv got EOF from 127.0.97.2:34649 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:14 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:14.892138 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:14.883965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:50973)
0926 23:24:14.884206 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:14.884225 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:14.884322 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:14.884808 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:14.884811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:14.885060 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:14.885198 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.885205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.886684 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.886688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.887561 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:14.887571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.888600 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.888604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:14.888797 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.889380 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:14.889396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:14.891942 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260926 23:24:14.892434 9501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:24:14.892697 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:14.884152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50973 (local address 127.0.97.62:44579)
0926 23:24:14.884437 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:14.884445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:14.884466 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:24:14.884544 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:14.884551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:14.884631 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:14.884791 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:14.885335 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.886540 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.887716 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:14.888468 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:14.888609 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:14.892191 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50973: BlockingRecv error: recv got EOF from 127.0.97.2:50973 (error 108)
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":147}
W20260926 23:24:15.187930 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:15.188194 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:15.188284 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:15.195266 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:15.195365 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:15.195475 9504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:24:15.205313 9504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36995
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:15.206465 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:15.207738 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:15.220253 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:15.220476 9512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:15.220232 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:15.221036 9504 server_base.cc:1034] running on GCE node
I20260926 23:24:15.221601 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:15.222512 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:15.223735 9504 hybrid_clock.cc:648] HybridClock initialized: now 1790465055223696 us; error 50 us; skew 500 ppm
I20260926 23:24:15.224052 9504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465055, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:15.227074 9504 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:24:15.228418 9504 webserver.cc:460] Webserver started at http://127.0.97.3:36995/ using document root <none> and password file <none>
I20260926 23:24:15.229136 9504 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:15.229301 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:15.234388 9504 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:24:15.237077 9519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:15.238416 9504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:24:15.238579 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "8676a2c9b83c419d80a08899018127e4"
format_stamp: "Formatted at 2026-09-26 23:24:12 on dist-test-slave-q5g8"
server_key: "2e9b635d8abe92bd44572e01b86701a3"
server_key_iv: "bfc307d0add28dacca44abaa986e4843"
server_key_version: "encryptionkey@0"
I20260926 23:24:15.239526 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:15.251168 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:15.255213 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:15.255507 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:15.256546 9504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:15.257951 9504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:15.258066 9504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:15.258195 9504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:15.258287 9504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:15.292557 9504 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36723
I20260926 23:24:15.292649 9631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36723 every 8 connection(s)
I20260926 23:24:15.294092 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:24:15.298621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9504
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:24:15.312914 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:15.296916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55237 (local address 127.0.97.62:44579)
0926 23:24:15.297114 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:15.297121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:15.298267 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 23:24:15.300976 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:15.300982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:15.301056 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:15.301154 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:15.302790 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.303757 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.306109 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.306584 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.306708 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.312584 (+ 5876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55237: BlockingRecv error: recv got EOF from 127.0.97.3:55237 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":66}
I20260926 23:24:15.313616 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:15.297334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:55237)
0926 23:24:15.298057 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:15.298106 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:15.300637 (+ 2531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:15.301380 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:15.301389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:15.302040 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:15.302631 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.302648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.303898 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.303904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.305944 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.305957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.306860 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.306869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.307214 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.307844 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:15.307876 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:15.312358 (+ 4482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":284,"threads_started":1}
W20260926 23:24:15.314544 9632 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44579 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44579: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260926 23:24:15.317113 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:15.299701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.0.97.62:44579)
0926 23:24:15.300351 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:15.300356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:15.300374 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:24:15.300436 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:15.300441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:15.300521 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:15.300658 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:15.302364 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.303472 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.305561 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.305926 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.306018 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.309705 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:15.309715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:15.309734 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:15.309779 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:15.312967 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:15.314418 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:15.314426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:15.314434 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:15.314643 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:15.315100 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:15.315108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:15.315113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:15.315674 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:15.315976 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:15.316353 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 23:24:15.316644 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":431,"threads_started":1}
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:15.323745 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:15.315219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:44847)
0926 23:24:15.315382 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:15.315405 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:15.315500 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:15.315806 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:15.315811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:15.316281 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:15.316476 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.316488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.317796 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.317802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.319047 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.319059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.319918 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.319923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.320091 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.320657 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:15.320704 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:15.323535 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 23:24:15.323897 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:15.315315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44847 (local address 127.0.97.62:44579)
0926 23:24:15.315472 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:15.315477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:15.315495 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:24:15.315596 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:15.315601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:15.315661 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:15.315756 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:15.316626 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.317656 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.319185 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.319736 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.319863 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.323688 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44847: BlockingRecv error: recv got EOF from 127.0.97.3:44847 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:15.331058 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:15.324399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:38139)
0926 23:24:15.324566 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:15.324589 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:15.324669 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:15.325000 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:15.325003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:15.325248 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:15.325400 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.325408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.326295 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.326299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.327185 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.327195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.327776 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.327780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.327908 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.328407 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:15.328423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:15.330870 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 23:24:15.331252 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:15.324540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38139 (local address 127.0.97.62:44579)
0926 23:24:15.324772 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:15.324776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:15.324789 (+ 13us) server_negotiation.cc:407] Connection header received
0926 23:24:15.324823 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:15.324826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:15.324883 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:15.324955 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:15.325516 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.326167 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.327297 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.327666 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.327749 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.331013 (+ 3264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38139: BlockingRecv error: recv got EOF from 127.0.97.3:38139 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260926 23:24:15.331365 9632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:15.462070 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:15.453911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:32905)
0926 23:24:15.454409 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:15.454425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:15.454535 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:15.454802 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:15.454805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:15.455248 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:15.455449 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.455458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.456497 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.456501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.457449 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.457461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.458131 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.458135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.458246 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.458768 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:15.458784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:15.461716 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":268,"threads_started":1}
W20260926 23:24:15.462128 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:15.453982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32905 (local address 127.0.97.62:44579)
0926 23:24:15.454153 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:15.454158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:15.454496 (+ 338us) server_negotiation.cc:407] Connection header received
0926 23:24:15.454623 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:15.454627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:15.454683 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:15.454764 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:15.455564 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.456366 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.457570 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.457992 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.458079 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.461901 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:32905: BlockingRecv error: recv got EOF from 127.0.97.1:32905 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:15 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:15.900827 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:15.893002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:59739)
0926 23:24:15.893517 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:15.893533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:15.893640 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:15.893912 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:15.893915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:15.894354 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:15.894543 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.894552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.895444 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.895448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.896399 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:15.896409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.897069 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.897073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:15.897197 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.897788 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:15.897804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:15.900520 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":294,"threads_started":1}
W20260926 23:24:15.901005 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:15.893104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59739 (local address 127.0.97.62:44579)
0926 23:24:15.893291 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:15.893295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:15.893606 (+ 311us) server_negotiation.cc:407] Connection header received
0926 23:24:15.893731 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:15.893734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:15.893791 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:15.893870 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:15.894659 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.895326 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.896522 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:15.896909 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:15.897002 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:15.900737 (+ 3735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59739: BlockingRecv error: recv got EOF from 127.0.97.2:59739 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 23:24:16 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:16 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:16.346864 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:16.332066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:56297)
0926 23:24:16.332709 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:16.332728 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:16.332910 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:16.333284 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:16.333288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:16.333822 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:16.334145 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:16.334156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.335103 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.335106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:16.336045 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:16.336056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.336834 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.336839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:16.336953 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:16.342962 (+ 6009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:16.342980 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:16.346544 (+ 3564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":363,"threads_started":1}
W20260926 23:24:16.346964 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:16.332129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56297 (local address 127.0.97.62:44579)
0926 23:24:16.332323 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:16.332332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:16.332803 (+ 471us) server_negotiation.cc:407] Connection header received
0926 23:24:16.333028 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:16.333034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:16.333105 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:16.333225 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:16.334270 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.334981 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.336190 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.336608 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.336862 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:16.346722 (+ 9860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56297: BlockingRecv error: recv got EOF from 127.0.97.3:56297 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 23:24:16 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:16 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:16.471401 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:16.462979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:34223)
0926 23:24:16.463503 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:16.463518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:16.463669 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:16.464052 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:16.464056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:16.464603 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:16.464891 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:16.464903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.466054 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.466058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:16.467061 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:16.467072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.467714 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.467719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:16.467871 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:16.468418 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:16.468437 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:16.471167 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":291,"threads_started":1}
W20260926 23:24:16.471606 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:16.463007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34223 (local address 127.0.97.62:44579)
0926 23:24:16.463182 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:16.463187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:16.463597 (+ 410us) server_negotiation.cc:407] Connection header received
0926 23:24:16.463786 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:16.463789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:16.463875 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:16.463983 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:16.465050 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.465941 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.467181 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.467583 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.467679 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:16.471359 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34223: BlockingRecv error: recv got EOF from 127.0.97.1:34223 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 23:24:16 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:16 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:16.910491 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:16.902073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:50947)
0926 23:24:16.902623 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:16.902638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:16.902759 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:16.903040 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:16.903043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:16.903543 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:16.903689 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:16.903698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.904848 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.904852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:16.905730 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:16.905741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.906499 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.906503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:16.906619 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:16.907242 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:16.907260 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:16.910111 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":324,"threads_started":1}
W20260926 23:24:16.910607 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:16.902093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50947 (local address 127.0.97.62:44579)
0926 23:24:16.902276 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:16.902281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:16.902711 (+ 430us) server_negotiation.cc:407] Connection header received
0926 23:24:16.902857 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:16.902860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:16.902918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:16.903002 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:16.903833 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.904728 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.905882 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:16.906336 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:16.906444 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:16.910334 (+ 3890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50947: BlockingRecv error: recv got EOF from 127.0.97.2:50947 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 23:24:17 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:17 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:17.356130 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:17.347973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:43637)
0926 23:24:17.348448 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:17.348465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:17.348578 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:17.348907 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:17.348910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:17.349304 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:17.349449 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:17.349457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.350429 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.350433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:17.351341 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:17.351352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.352120 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.352127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:17.352343 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:17.353084 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:17.353103 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:17.355864 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":237,"threads_started":1}
W20260926 23:24:17.356372 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:17.348236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43637 (local address 127.0.97.62:44579)
0926 23:24:17.348451 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:17.348456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:17.348537 (+ 81us) server_negotiation.cc:407] Connection header received
0926 23:24:17.348673 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:17.348694 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:17.348765 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:17.348849 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:17.349574 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.350271 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.351461 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.351939 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.352067 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:17.356055 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43637: BlockingRecv error: recv got EOF from 127.0.97.3:43637 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 23:24:17 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:17 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:17.480870 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:17.472226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:57191)
0926 23:24:17.472812 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:17.472828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:17.472990 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:17.473317 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:17.473322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:17.473696 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:17.473865 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:17.473874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.474929 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.474937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:17.475842 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:17.475853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.476538 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.476544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:17.476768 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:17.477471 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:17.477489 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:17.480552 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":304,"threads_started":1}
W20260926 23:24:17.481033 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:17.472393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57191 (local address 127.0.97.62:44579)
0926 23:24:17.472641 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:17.472649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:17.472919 (+ 270us) server_negotiation.cc:407] Connection header received
0926 23:24:17.473120 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:17.473129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:17.473189 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:17.473290 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:17.474013 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.474789 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.475997 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.476390 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.476526 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:17.480785 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57191: BlockingRecv error: recv got EOF from 127.0.97.1:57191 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 23:24:17 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:17 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:17.920137 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:17.911717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:56199)
0926 23:24:17.912207 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:17.912223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:17.912382 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:17.912731 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:17.912737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:17.913094 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:17.913255 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:17.913264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.914249 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.914256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:17.915151 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:17.915161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.915851 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.915856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:17.916036 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:17.916710 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:17.916728 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:17.919730 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":301,"threads_started":1}
W20260926 23:24:17.920243 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:17.911767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56199 (local address 127.0.97.62:44579)
0926 23:24:17.911968 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:17.911976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:17.912315 (+ 339us) server_negotiation.cc:407] Connection header received
0926 23:24:17.912502 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:17.912507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:17.912576 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:17.912678 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:17.913395 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.914126 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.915295 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:17.915700 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:17.915840 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:17.919958 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56199: BlockingRecv error: recv got EOF from 127.0.97.2:56199 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 23:24:18 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:18 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:18.366442 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:18.357474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:59797)
0926 23:24:18.357996 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:18.358020 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:18.358146 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:18.358549 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:18.358552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:18.358912 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:18.359077 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:18.359085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.360351 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.360358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:18.361407 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:18.361420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.362153 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.362159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:18.362340 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:18.363059 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:18.363077 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:18.366065 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260926 23:24:18.366559 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:18.357602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59797 (local address 127.0.97.62:44579)
0926 23:24:18.357811 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:18.357815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:18.358119 (+ 304us) server_negotiation.cc:407] Connection header received
0926 23:24:18.358298 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:18.358303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:18.358388 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:18.358509 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:18.359233 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.360169 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.361558 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.361981 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.362119 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:18.366298 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59797: BlockingRecv error: recv got EOF from 127.0.97.3:59797 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 23:24:18 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:18 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:18.491101 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:18.481982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:47669)
0926 23:24:18.482565 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:18.482588 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:18.482711 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:18.483181 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:18.483184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:18.483546 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:18.483748 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:18.483757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.484940 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.484945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:18.486354 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:18.486400 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.487193 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.487198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:18.487344 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:18.487951 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:18.487969 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:18.490874 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":351,"threads_started":1}
W20260926 23:24:18.491310 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:18.482053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47669 (local address 127.0.97.62:44579)
0926 23:24:18.482256 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:18.482260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:18.482666 (+ 406us) server_negotiation.cc:407] Connection header received
0926 23:24:18.482811 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:18.482815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:18.482877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:18.483138 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:18.483883 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.484756 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.486567 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.487006 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.487106 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:18.491081 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47669: BlockingRecv error: recv got EOF from 127.0.97.1:47669 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 23:24:18 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:18 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:18.932075 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:18.921588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:33033)
0926 23:24:18.922116 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:18.922133 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:18.922260 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:18.922668 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:18.922671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:18.923138 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:18.923347 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:18.923360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.924895 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.924900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:18.926548 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:18.926565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.927379 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.927386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:18.927601 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:18.928285 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:18.928309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:18.931750 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
W20260926 23:24:18.932214 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:18.921632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33033 (local address 127.0.97.62:44579)
0926 23:24:18.921837 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:18.921844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:18.922227 (+ 383us) server_negotiation.cc:407] Connection header received
0926 23:24:18.922376 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:18.922382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:18.922483 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:18.922627 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:18.923555 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.924723 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.926739 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:18.927215 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:18.927326 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:18.931990 (+ 4664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33033: BlockingRecv error: recv got EOF from 127.0.97.2:33033 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 23:24:19 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:19 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:19.375507 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:19.367462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:46733)
0926 23:24:19.367916 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:19.367932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:19.368049 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:19.368370 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:19.368374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:19.368785 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:19.368959 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:19.368968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.369969 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.369972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:19.370949 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:19.370961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.371641 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.371645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:19.371772 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:19.372349 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:19.372367 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:19.375262 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":255,"threads_started":1}
W20260926 23:24:19.375715 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:19.367615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46733 (local address 127.0.97.62:44579)
0926 23:24:19.367843 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:19.367849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:19.368010 (+ 161us) server_negotiation.cc:407] Connection header received
0926 23:24:19.368161 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:19.368164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:19.368240 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:19.368323 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:19.369093 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.369856 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.371090 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.371500 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.371597 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:19.375464 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46733: BlockingRecv error: recv got EOF from 127.0.97.3:46733 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 23:24:19 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:19 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:19.501760 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:19.492344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:56905)
0926 23:24:19.493020 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:19.493043 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:19.493186 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:19.493680 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:19.493684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:19.494200 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:19.494365 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:19.494375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.495445 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.495449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:19.496433 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:19.496447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.497193 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.497199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:19.497371 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:19.498129 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:19.498148 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:19.501369 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":359,"threads_started":1}
W20260926 23:24:19.501819 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:19.492449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56905 (local address 127.0.97.62:44579)
0926 23:24:19.492705 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:19.492715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:19.493337 (+ 622us) server_negotiation.cc:407] Connection header received
0926 23:24:19.493400 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:19.493405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:19.493480 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:19.493582 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:19.494526 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.495280 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.496588 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.497013 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.497118 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:19.501593 (+ 4475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56905: BlockingRecv error: recv got EOF from 127.0.97.1:56905 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 23:24:19 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:19 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:19.942619 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:19.933364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:58725)
0926 23:24:19.933979 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:19.934002 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:19.934199 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:19.934681 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:19.934687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:19.935285 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:19.935547 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:19.935561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.936738 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.936744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:19.937699 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:19.937712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.938464 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.938470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:19.938668 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:19.939290 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:19.939308 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:19.942334 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":283,"threads_started":1}
W20260926 23:24:19.942811 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:19.933678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58725 (local address 127.0.97.62:44579)
0926 23:24:19.933932 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:19.933941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:19.934111 (+ 170us) server_negotiation.cc:407] Connection header received
0926 23:24:19.934374 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:19.934378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:19.934446 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:19.934561 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:19.935680 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.936548 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.937858 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:19.938300 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:19.938438 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:19.942526 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58725: BlockingRecv error: recv got EOF from 127.0.97.2:58725 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 26 23:24:20 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:20 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:20.385339 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:20.376521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:58427)
0926 23:24:20.377174 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:20.377191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:20.377361 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:20.377729 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:20.377734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:20.378250 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:20.378425 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:20.378434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.379468 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.379475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:20.380420 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:20.380431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.381215 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.381220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:20.381396 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:20.381979 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:20.381996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:20.384985 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":321,"threads_started":1}
W20260926 23:24:20.385434 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:20.376759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58427 (local address 127.0.97.62:44579)
0926 23:24:20.376967 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:20.376972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:20.377291 (+ 319us) server_negotiation.cc:407] Connection header received
0926 23:24:20.377492 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:20.377498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:20.377587 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:20.377697 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:20.378567 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.379320 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.380566 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.381069 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.381204 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:20.385176 (+ 3972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58427: BlockingRecv error: recv got EOF from 127.0.97.3:58427 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 23:24:20 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:20 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:20.512259 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:20.503011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:48131)
0926 23:24:20.503711 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:20.503729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:20.503899 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:20.504343 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:20.504351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:20.504775 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:20.504933 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:20.504946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.506080 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.506084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:20.507023 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:20.507034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.507893 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.507904 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:20.508068 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:20.508765 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:20.508784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:20.512015 (+ 3231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":380,"threads_started":1}
W20260926 23:24:20.512637 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:20.503095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48131 (local address 127.0.97.62:44579)
0926 23:24:20.503339 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:20.503348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:20.503828 (+ 480us) server_negotiation.cc:407] Connection header received
0926 23:24:20.504052 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:20.504056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:20.504125 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:20.504257 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:20.505100 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.505934 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.507164 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.507683 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.507794 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:20.512275 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48131: BlockingRecv error: recv got EOF from 127.0.97.1:48131 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 23:24:20 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:20 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:20.952601 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:20.943685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:56019)
0926 23:24:20.944264 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:20.944282 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:20.944422 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:20.944858 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:20.944862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:20.945235 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:20.945382 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:20.945391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.946929 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.946934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:20.947994 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:20.948008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.948665 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.948670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:20.948853 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:20.949455 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:20.949473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:20.952358 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":304,"threads_started":1}
W20260926 23:24:20.953052 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:20.943806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56019 (local address 127.0.97.62:44579)
0926 23:24:20.943984 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:20.943990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:20.944371 (+ 381us) server_negotiation.cc:407] Connection header received
0926 23:24:20.944559 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:20.944563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:20.944629 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:20.944825 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:20.945614 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.946750 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.948125 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:20.948529 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:20.948634 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:20.952763 (+ 4129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56019: BlockingRecv error: recv got EOF from 127.0.97.2:56019 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 23:24:21 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:21 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:21.399676 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:21.386386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:46151)
0926 23:24:21.386916 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:21.386932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:21.387089 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:21.387475 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:21.387479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:21.387847 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:21.388026 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:21.388039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.389437 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.389441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:21.390379 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:21.390391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.391133 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.391139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:21.391270 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:21.396415 (+ 5145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:21.396437 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:21.399443 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":298,"threads_started":1}
W20260926 23:24:21.400202 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:21.386480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46151 (local address 127.0.97.62:44579)
0926 23:24:21.386719 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:21.386728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:21.387019 (+ 291us) server_negotiation.cc:407] Connection header received
0926 23:24:21.387227 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:21.387232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:21.387304 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:21.387445 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:21.388169 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.389269 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.390530 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.390962 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.391073 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:21.399750 (+ 8677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46151: BlockingRecv error: recv got EOF from 127.0.97.3:46151 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 23:24:21 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:21 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:21.522687 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:21.513473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:57061)
0926 23:24:21.514119 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:21.514136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:21.514261 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:21.514645 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:21.514650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:21.515140 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:21.515314 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:21.515323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.516532 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.516536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:21.517623 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:21.517635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.518383 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.518389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:21.518569 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:21.519221 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:21.519239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:21.522283 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":281,"threads_started":1}
W20260926 23:24:21.522706 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:21.513590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57061 (local address 127.0.97.62:44579)
0926 23:24:21.513798 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:21.513803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:21.514222 (+ 419us) server_negotiation.cc:407] Connection header received
0926 23:24:21.514405 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:21.514409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:21.514476 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:21.514575 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:21.515477 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.516360 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.517774 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.518212 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.518347 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:21.522492 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57061: BlockingRecv error: recv got EOF from 127.0.97.1:57061 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 23:24:21 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:21 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:21.964363 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:21.953827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:46199)
0926 23:24:21.954537 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:21.954555 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:21.954693 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:21.955201 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:21.955205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:21.955662 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:21.955875 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:21.955886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.957203 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.957208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:21.958265 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:21.958279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.959062 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.959067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:21.959189 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:21.960247 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:21.960310 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:21.964090 (+ 3780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":303,"threads_started":1}
W20260926 23:24:21.964644 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:21.953963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46199 (local address 127.0.97.62:44579)
0926 23:24:21.954226 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:21.954231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:21.954676 (+ 445us) server_negotiation.cc:407] Connection header received
0926 23:24:21.954867 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:21.954877 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:21.954991 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:21.955109 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:21.956060 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.957040 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.958440 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:21.958893 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:21.959012 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:21.964375 (+ 5363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46199: BlockingRecv error: recv got EOF from 127.0.97.2:46199 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Sep 26 23:24:22 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:22 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:22.411360 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:22.400987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:53767)
0926 23:24:22.401984 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:22.402011 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:22.402166 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:22.402644 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:22.402649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:22.403424 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:22.403742 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:22.403760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.405269 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.405274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:22.406273 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:22.406287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.407091 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.407096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:22.407251 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:22.407929 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:22.407947 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:22.411110 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":440,"threads_started":1}
W20260926 23:24:22.411578 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:22.401854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53767 (local address 127.0.97.62:44579)
0926 23:24:22.402133 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:22.402141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:22.402173 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:24:22.402346 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:22.402353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:22.402452 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:22.402594 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:22.403960 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.405076 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.406438 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.406909 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.407017 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:22.411338 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53767: BlockingRecv error: recv got EOF from 127.0.97.3:53767 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 26 23:24:22 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:22 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:22.533401 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:22.523981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:41523)
0926 23:24:22.524669 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:22.524711 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:22.524850 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:22.525229 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:22.525233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:22.525666 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:22.525822 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:22.525830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.526863 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.526867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:22.527850 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:22.527862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.528846 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.528852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:22.528970 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:22.529746 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:22.529774 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:22.533058 (+ 3284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":397,"threads_started":1}
W20260926 23:24:22.533553 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:22.524237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41523 (local address 127.0.97.62:44579)
0926 23:24:22.524453 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:22.524458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:22.524836 (+ 378us) server_negotiation.cc:407] Connection header received
0926 23:24:22.524970 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:22.524974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:22.525041 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:22.525149 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:22.525961 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.526702 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.528017 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.528531 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.528665 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:22.533296 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41523: BlockingRecv error: recv got EOF from 127.0.97.1:41523 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 23:24:22 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:22 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:22.974925 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:22.965690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:33219)
0926 23:24:22.966204 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:22.966231 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:22.966385 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:22.966766 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:22.966771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:22.967308 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:22.967507 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:22.967517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.968790 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.968797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:22.969840 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:22.969853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.970647 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.970653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:22.970794 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:22.971471 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:22.971489 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:22.974668 (+ 3179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":280,"threads_started":1}
W20260926 23:24:22.975142 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:22.965854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33219 (local address 127.0.97.62:44579)
0926 23:24:22.966070 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:22.966076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:22.966353 (+ 277us) server_negotiation.cc:407] Connection header received
0926 23:24:22.966501 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:22.966505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:22.966605 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:22.966701 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:22.967674 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.968568 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.970005 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:22.970446 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:22.970556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:22.974913 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33219: BlockingRecv error: recv got EOF from 127.0.97.2:33219 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 23:24:23 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:23 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:23.422767 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:23.412452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:45031)
0926 23:24:23.413111 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:23.413129 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:23.413340 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:23.413727 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:23.413732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:23.414188 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:23.414343 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:23.414353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.415438 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.415443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:23.416735 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:23.416754 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.417792 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.417800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:23.418009 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:23.418649 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:23.418667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:23.422512 (+ 3845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":372,"threads_started":1}
W20260926 23:24:23.423169 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:23.412597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45031 (local address 127.0.97.62:44579)
0926 23:24:23.412878 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:23.412887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:23.413261 (+ 374us) server_negotiation.cc:407] Connection header received
0926 23:24:23.413485 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:23.413489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:23.413565 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:23.413674 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:23.414506 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.415270 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.416916 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.417562 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.417705 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:23.422826 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45031: BlockingRecv error: recv got EOF from 127.0.97.3:45031 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 23:24:23 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:23 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:23.543622 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:23.534397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:48881)
0926 23:24:23.535027 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:23.535042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:23.535232 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:23.535556 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:23.535561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:23.535996 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:23.536164 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:23.536180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.537278 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.537285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:23.538262 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:23.538274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.539027 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.539034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:23.539227 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:23.539864 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:23.539882 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:23.543369 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":392,"threads_started":1}
W20260926 23:24:23.543910 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:23.534488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48881 (local address 127.0.97.62:44579)
0926 23:24:23.534843 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:23.534851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:23.535125 (+ 274us) server_negotiation.cc:407] Connection header received
0926 23:24:23.535366 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:23.535370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:23.535430 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:23.535527 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:23.536315 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.537151 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.538411 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.538862 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.539027 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:23.543623 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48881: BlockingRecv error: recv got EOF from 127.0.97.1:48881 (error 108)
Metrics: {"server-negotiator.queue_time_us":193}
Sep 26 23:24:23 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:23 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:23.985357 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:23.976005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:35751)
0926 23:24:23.976620 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:23.976638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:23.976800 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:23.977106 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:23.977109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:23.977504 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:23.977691 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:23.977700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.978863 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.978871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:23.980029 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:23.980045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.981048 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.981057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:23.981275 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:23.981947 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:23.981966 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:23.985087 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":334,"threads_started":1}
W20260926 23:24:23.985527 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:23.976106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35751 (local address 127.0.97.62:44579)
0926 23:24:23.976378 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:23.976383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:23.976741 (+ 358us) server_negotiation.cc:407] Connection header received
0926 23:24:23.976903 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:23.976907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:23.976969 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:23.977059 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:23.977828 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.978712 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.980236 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:23.980882 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:23.981017 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:23.985277 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35751: BlockingRecv error: recv got EOF from 127.0.97.2:35751 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 26 23:24:24 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:24 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:24.434177 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:24.424378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:35873)
0926 23:24:24.425177 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:24.425199 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:24.425352 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:24.425703 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:24.425708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:24.426253 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:24.426492 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:24.426507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.428119 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.428123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:24.429153 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:24.429166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.429989 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.429994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:24.430128 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:24.430742 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:24.430760 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:24.433775 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":477,"threads_started":1}
W20260926 23:24:24.434266 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:24.424448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35873 (local address 127.0.97.62:44579)
0926 23:24:24.424645 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:24.424650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:24.425285 (+ 635us) server_negotiation.cc:407] Connection header received
0926 23:24:24.425485 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:24.425491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:24.425569 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:24.425669 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:24.426691 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.427949 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.429406 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.429831 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.429947 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:24.434027 (+ 4080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35873: BlockingRecv error: recv got EOF from 127.0.97.3:35873 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 23:24:24 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:24 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:24.554179 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:24.544778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:44641)
0926 23:24:24.545320 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:24.545339 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:24.545475 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:24.545983 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:24.545987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:24.546573 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:24.546736 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:24.546746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.548284 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.548288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:24.549315 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:24.549328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.550096 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.550101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:24.550218 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:24.550903 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:24.550922 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:24.553851 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":328,"threads_started":1}
W20260926 23:24:24.554474 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:24.544877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44641 (local address 127.0.97.62:44579)
0926 23:24:24.545116 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:24.545126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:24.545437 (+ 311us) server_negotiation.cc:407] Connection header received
0926 23:24:24.545648 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:24.545653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:24.545724 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:24.545825 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:24.546884 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.548118 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.549448 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.549910 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.550011 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:24.554068 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44641: BlockingRecv error: recv got EOF from 127.0.97.1:44641 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 23:24:24 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:24 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:24.995535 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:24.986436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:44487)
0926 23:24:24.987099 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:24.987122 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:24.987274 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:24.987577 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:24.987581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:24.987945 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:24.988141 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:24.988150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.989173 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.989178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:24.990171 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:24.990185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.991153 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.991161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:24.991340 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:24.992075 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:24.992101 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:24.995300 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":306,"threads_started":1}
W20260926 23:24:24.995811 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:24.986533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44487 (local address 127.0.97.62:44579)
0926 23:24:24.986714 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:24.986719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:24.987211 (+ 492us) server_negotiation.cc:407] Connection header received
0926 23:24:24.987379 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:24.987383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:24.987446 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:24.987534 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:24.988275 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.989022 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.990380 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:24.990890 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:24.991017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:24.995519 (+ 4502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44487: BlockingRecv error: recv got EOF from 127.0.97.2:44487 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 23:24:25 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:25 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:25.444631 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:25.435235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:57215)
0926 23:24:25.436036 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:25.436055 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:25.436201 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:25.436585 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:25.436593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:25.437045 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:25.437270 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:25.437283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.438445 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.438450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:25.439518 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:25.439532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.440309 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.440314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:25.440440 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:25.441160 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:25.441180 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:25.444293 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":588,"threads_started":1}
W20260926 23:24:25.444753 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:25.435536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57215 (local address 127.0.97.62:44579)
0926 23:24:25.435760 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:25.435768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:25.436156 (+ 388us) server_negotiation.cc:407] Connection header received
0926 23:24:25.436338 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:25.436341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:25.436448 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:25.436530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:25.437432 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.438293 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.439686 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.440136 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.440255 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:25.444484 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57215: BlockingRecv error: recv got EOF from 127.0.97.3:57215 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 23:24:25 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:25 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:25.564311 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:25.555145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:46737)
0926 23:24:25.555649 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:25.555665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:25.555786 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:25.556154 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:25.556157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:25.556566 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:25.556783 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:25.556792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.557880 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.557885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:25.558797 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:25.558810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.559828 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.559835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:25.560040 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:25.560773 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:25.560796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:25.564040 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":287,"threads_started":1}
W20260926 23:24:25.564505 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:25.555249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46737 (local address 127.0.97.62:44579)
0926 23:24:25.555481 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:25.555488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:25.555749 (+ 261us) server_negotiation.cc:407] Connection header received
0926 23:24:25.555888 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:25.555892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:25.556018 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:25.556109 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:25.557001 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.557727 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.558990 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.559573 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.559723 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:25.564287 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46737: BlockingRecv error: recv got EOF from 127.0.97.1:46737 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 23:24:25 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:25 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:26.005728 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:25.996463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:44149)
0926 23:24:25.997052 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:25.997069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:25.997182 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:25.997494 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:25.997498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:25.997938 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:25.998110 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:25.998119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.999153 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:25.999157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:26.000047 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:26.000187 (+ 140us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.001114 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.001119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:26.001259 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:26.002042 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:26.002061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:26.005341 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":349,"threads_started":1}
W20260926 23:24:26.005815 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:25.996550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44149 (local address 127.0.97.62:44579)
0926 23:24:25.996760 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:25.996766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:25.997160 (+ 394us) server_negotiation.cc:407] Connection header received
0926 23:24:25.997276 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:25.997280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:25.997342 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:25.997443 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:25.998243 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:25.998994 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.000336 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.000959 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.001068 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:26.005574 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44149: BlockingRecv error: recv got EOF from 127.0.97.2:44149 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 23:24:26 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:26 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:26.461585 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:26.445721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:60637)
0926 23:24:26.446266 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:26.446282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:26.446405 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:26.446777 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:26.446781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:26.447264 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:26.447470 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:26.447479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.448479 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.448483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:26.449479 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:26.449493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.450287 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.450291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:26.450438 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:26.458408 (+ 7970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:26.458431 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:26.461310 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":299,"threads_started":1}
W20260926 23:24:26.461797 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:26.445821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60637 (local address 127.0.97.62:44579)
0926 23:24:26.445989 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:26.445994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:26.446350 (+ 356us) server_negotiation.cc:407] Connection header received
0926 23:24:26.446512 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:26.446516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:26.446577 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:26.446664 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:26.447641 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.448362 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.449643 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.450134 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.450241 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:26.461573 (+ 11332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60637: BlockingRecv error: recv got EOF from 127.0.97.3:60637 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 23:24:26 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:26 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:26.574357 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:26.565574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:53771)
0926 23:24:26.566124 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:26.566141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:26.566269 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:26.566611 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:26.566614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:26.567000 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:26.567228 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:26.567240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.568496 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.568502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:26.569816 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:26.569830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.570556 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.570560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:26.570698 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:26.571286 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:26.571304 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:26.574119 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
W20260926 23:24:26.574568 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:26.565621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53771 (local address 127.0.97.62:44579)
0926 23:24:26.565854 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:26.565864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:26.566244 (+ 380us) server_negotiation.cc:407] Connection header received
0926 23:24:26.566390 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:26.566394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:26.566478 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:26.566576 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:26.567423 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.568351 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.569955 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:26.570424 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:26.570531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:26.574331 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53771: BlockingRecv error: recv got EOF from 127.0.97.1:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 23:24:27 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:27 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:27.015236 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:27.006821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:60555)
0926 23:24:27.007319 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:27.007336 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:27.007460 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:27.007752 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:27.007755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:27.008148 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:27.008304 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:27.008312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.009339 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.009343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:27.010328 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:27.010342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.011016 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.011020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:27.011148 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:27.011825 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:27.011842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:27.014898 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":287,"threads_started":1}
W20260926 23:24:27.015511 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:27.006881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60555 (local address 127.0.97.62:44579)
0926 23:24:27.007139 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:27.007145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:27.007422 (+ 277us) server_negotiation.cc:407] Connection header received
0926 23:24:27.007557 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:27.007560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:27.007622 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:27.007705 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:27.008459 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.009210 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.010461 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.010870 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.010967 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:27.015199 (+ 4232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60555: BlockingRecv error: recv got EOF from 127.0.97.2:60555 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 23:24:27 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:27 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:27.472251 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:27.462682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:57509)
0926 23:24:27.463367 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:27.463386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:27.463574 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:27.464004 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:27.464009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:27.464452 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:27.464629 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:27.464639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.465936 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.465941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:27.467098 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:27.467115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.468112 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.468117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:27.468271 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:27.469066 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:27.469086 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:27.472006 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":369,"threads_started":1}
W20260926 23:24:27.472587 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:27.462904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57509 (local address 127.0.97.62:44579)
0926 23:24:27.463195 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:27.463203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:27.463491 (+ 288us) server_negotiation.cc:407] Connection header received
0926 23:24:27.463719 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:27.463730 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:27.463796 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:27.463956 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:27.464861 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.465740 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.467275 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.467954 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.468055 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:27.472296 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57509: BlockingRecv error: recv got EOF from 127.0.97.3:57509 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 26 23:24:27 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:27 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:27.585423 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:27.575513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:58729)
0926 23:24:27.576030 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:27.576046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:27.576174 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:27.576515 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:27.576518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:27.576971 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:27.577185 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:27.577195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.578456 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.578460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:27.579786 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:27.579802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.580772 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.580780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:27.580958 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:27.581675 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:27.581695 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:27.585143 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":268,"threads_started":1}
W20260926 23:24:27.585655 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:27.575675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58729 (local address 127.0.97.62:44579)
0926 23:24:27.575952 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:27.575958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:27.576146 (+ 188us) server_negotiation.cc:407] Connection header received
0926 23:24:27.576294 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:27.576298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:27.576366 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:27.576452 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:27.577347 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.578307 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.579951 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:27.580502 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:27.580644 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:27.585380 (+ 4736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58729: BlockingRecv error: recv got EOF from 127.0.97.1:58729 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 26 23:24:28 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:28 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:28.025552 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:28.016736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:44403)
0926 23:24:28.017352 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:28.017370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:28.017496 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:28.017914 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:28.017918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:28.018362 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:28.018548 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:28.018558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.019790 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.019794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:28.020860 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:28.020874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.021639 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.021645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:28.021770 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:28.022467 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:28.022485 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:28.025318 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":327,"threads_started":1}
W20260926 23:24:28.025976 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:28.016940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44403 (local address 127.0.97.62:44579)
0926 23:24:28.017106 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:28.017111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:28.017473 (+ 362us) server_negotiation.cc:407] Connection header received
0926 23:24:28.017648 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:28.017652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:28.017766 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:28.017863 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:28.018719 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.019578 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.020995 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.021444 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.021543 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:28.025573 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44403: BlockingRecv error: recv got EOF from 127.0.97.2:44403 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 23:24:28 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:28 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:28.483845 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:28.473381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:50843)
0926 23:24:28.473963 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:28.473980 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:28.474102 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:28.474446 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:28.474450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:28.474838 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:28.474995 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:28.475003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.476129 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.476133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:28.477656 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:28.477715 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.478583 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.478590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:28.478801 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:28.479743 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:28.479765 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:28.483372 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
W20260926 23:24:28.483969 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:28.473549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50843 (local address 127.0.97.62:44579)
0926 23:24:28.473776 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:28.473781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:28.474088 (+ 307us) server_negotiation.cc:407] Connection header received
0926 23:24:28.474226 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:28.474230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:28.474298 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:28.474393 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:28.475137 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.475954 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.477874 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.478376 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.478561 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:28.483660 (+ 5099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50843: BlockingRecv error: recv got EOF from 127.0.97.3:50843 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 23:24:28 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:28 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:28.595824 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:28.586459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:40311)
0926 23:24:28.587119 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:28.587153 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:28.587297 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:28.587755 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:28.587760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:28.588157 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:28.588367 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:28.588378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.589938 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.589943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:28.591095 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:28.591112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.591915 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.591920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:28.592049 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:28.592665 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:28.592739 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:28.595557 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":422,"threads_started":1}
W20260926 23:24:28.596138 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:28.586634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40311 (local address 127.0.97.62:44579)
0926 23:24:28.587007 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:28.587020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:28.587254 (+ 234us) server_negotiation.cc:407] Connection header received
0926 23:24:28.587458 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:28.587469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:28.587542 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:28.587655 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:28.588565 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.589787 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.591281 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:28.591771 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:28.591881 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:28.595828 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40311: BlockingRecv error: recv got EOF from 127.0.97.1:40311 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 26 23:24:29 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:29 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:29.035912 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:29.026583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:57631)
0926 23:24:29.027110 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:29.027129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:29.027290 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:29.027618 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:29.027622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:29.028043 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:29.028251 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:29.028259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.029402 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.029408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:29.030351 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:29.030363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.031094 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.031100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:29.031283 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:29.031930 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:29.031947 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:29.035519 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"spinlock_wait_cycles":25344,"thread_start_us":354,"threads_started":1}
W20260926 23:24:29.035991 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:29.026762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57631 (local address 127.0.97.62:44579)
0926 23:24:29.026985 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:29.026991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:29.027214 (+ 223us) server_negotiation.cc:407] Connection header received
0926 23:24:29.027409 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:29.027415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:29.027485 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:29.027586 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:29.028406 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.029240 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.030498 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.030942 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.031076 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:29.035778 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57631: BlockingRecv error: recv got EOF from 127.0.97.2:57631 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 23:24:29 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:29 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:29.494515 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:29.484915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:45269)
0926 23:24:29.485486 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:29.485505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:29.485647 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:29.486024 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:29.486028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:29.486422 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:29.486630 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:29.486641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.487883 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.487888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:29.489288 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:29.489304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.490144 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.490149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:29.490297 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:29.491051 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:29.491072 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:29.494257 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":321,"threads_started":1}
W20260926 23:24:29.494959 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:29.485011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45269 (local address 127.0.97.62:44579)
0926 23:24:29.485229 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:29.485235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:29.485627 (+ 392us) server_negotiation.cc:407] Connection header received
0926 23:24:29.485759 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:29.485763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:29.485830 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:29.485927 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:29.486823 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.487671 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.489468 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.489970 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.490094 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:29.494662 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45269: BlockingRecv error: recv got EOF from 127.0.97.3:45269 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 23:24:29 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:29 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:29.605928 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:29.597157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:52841)
0926 23:24:29.597716 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:29.597732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:29.597882 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:29.598217 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:29.598222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:29.598682 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:29.598847 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:29.598856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.599876 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.599882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:29.600852 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:29.600864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.601712 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.601718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:29.601883 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:29.602520 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:29.602538 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:29.605591 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":306,"threads_started":1}
W20260926 23:24:29.606019 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:29.597272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52841 (local address 127.0.97.62:44579)
0926 23:24:29.597531 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:29.597538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:29.597822 (+ 284us) server_negotiation.cc:407] Connection header received
0926 23:24:29.598009 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:29.598014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:29.598088 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:29.598187 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:29.598992 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.599719 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.601020 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:29.601569 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:29.601700 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:29.605806 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52841: BlockingRecv error: recv got EOF from 127.0.97.1:52841 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 23:24:30 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:24:30 dist-test-slave-q5g8 krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:24:30.046465 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:30.037078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:58667)
0926 23:24:30.037646 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:30.037665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:30.037782 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:30.038082 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:30.038085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:30.038481 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:30.038658 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:30.038666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.039807 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.039812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:30.040909 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:30.040924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.041700 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.041705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:30.041824 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:30.042497 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:30.042517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:30.046062 (+ 3545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44579: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":335,"threads_started":1}
W20260926 23:24:30.046618 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:30.037261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58667 (local address 127.0.97.62:44579)
0926 23:24:30.037438 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:30.037443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:30.037767 (+ 324us) server_negotiation.cc:407] Connection header received
0926 23:24:30.037883 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:30.037887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:30.037951 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:30.038050 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:30.038815 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.039596 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.041080 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.041535 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.041644 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:30.046323 (+ 4679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58667: BlockingRecv error: recv got EOF from 127.0.97.2:58667 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:24:30.308874 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9222
I20260926 23:24:30.323856 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:30.309937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.0.97.1:42621)
0926 23:24:30.310459 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:30.310467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:30.310495 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:24:30.310556 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:30.310562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:30.310831 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:30.311013 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:30.312408 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.313518 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.314901 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.315485 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.315618 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:30.319027 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:30.319043 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:30.319069 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:30.319125 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:30.323048 (+ 3923us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:24:30.323493 (+ 445us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37362: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
W20260926 23:24:30.323998 9682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37362: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 23:24:30.324030 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9222: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:42621: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:24:30.324237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9222
I20260926 23:24:30.342516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9373
I20260926 23:24:30.355257 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:30.343675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.97.2:44761)
0926 23:24:30.344235 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:30.344242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:30.344273 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:24:30.344331 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:30.344337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:30.344557 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:30.344805 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:30.346118 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.347204 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.348537 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.349227 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.349373 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:30.351904 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:30.351914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:30.351929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:30.351974 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:30.354509 (+ 2535us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:24:30.354853 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51314: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":215,"threads_started":1}
W20260926 23:24:30.355255 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9373: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:44761: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 23:24:30.355372 9683 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51314: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:24:30.355428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9373
I20260926 23:24:30.373178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9504
I20260926 23:24:30.386480 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:30.374186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.0.97.3:36723)
0926 23:24:30.374835 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:30.374842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:30.374870 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:24:30.374931 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:30.374938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:30.375171 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:30.375334 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:30.376793 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.377912 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.379261 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:30.379960 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:30.380071 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:30.382746 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:30.382755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:30.382772 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:30.382833 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:30.385862 (+ 3029us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:24:30.386174 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36070: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":366,"threads_started":1}
W20260926 23:24:30.386610 9684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36070: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 23:24:30.386708 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9504: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36723: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:24:30.386886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9504
I20260926 23:24:30.403983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9153
I20260926 23:24:30.405400 9215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:30.526620 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9153
2026-09-26T23:24:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21134 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T23:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:24:30Z Disabled control of system clock
I20260926 23:24:30.605221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35805
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 23:24:30.948856 9689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:30.949175 9689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:30.949255 9689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:30.956753 9689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:24:30.956866 9689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:30.956907 9689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:30.956956 9689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:24:30.957002 9689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:24:30.966615 9689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:30.967937 9689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:30.969785 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:30.983395 9694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:30.983419 9697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:30.983373 9695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:30.983892 9689 server_base.cc:1034] running on GCE node
I20260926 23:24:30.985100 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:30.986662 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:30.987903 9689 hybrid_clock.cc:648] HybridClock initialized: now 1790465070987858 us; error 37 us; skew 500 ppm
I20260926 23:24:30.988327 9689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:30.991047 9689 webserver.cc:460] Webserver started at http://127.0.97.62:34827/ using document root <none> and password file <none>
I20260926 23:24:30.991786 9689 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:30.991904 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:30.992339 9689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:30.995682 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "046ecef4075a45f89a981ba9ef399783"
format_stamp: "Formatted at 2026-09-26 23:24:30 on dist-test-slave-q5g8"
server_key: "1d2f441557939acf6aa3db5953d9aa3c"
server_key_iv: "747024229b24d1b4de2456d1e66e63fc"
server_key_version: "encryptionkey@0"
I20260926 23:24:30.996577 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "046ecef4075a45f89a981ba9ef399783"
format_stamp: "Formatted at 2026-09-26 23:24:30 on dist-test-slave-q5g8"
server_key: "1d2f441557939acf6aa3db5953d9aa3c"
server_key_iv: "747024229b24d1b4de2456d1e66e63fc"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.001972 9689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 23:24:31.005710 9703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:31.007129 9689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 23:24:31.007320 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "046ecef4075a45f89a981ba9ef399783"
format_stamp: "Formatted at 2026-09-26 23:24:30 on dist-test-slave-q5g8"
server_key: "1d2f441557939acf6aa3db5953d9aa3c"
server_key_iv: "747024229b24d1b4de2456d1e66e63fc"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.007561 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:31.028120 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:31.029114 9689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:31.029423 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:31.048369 9689 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35805
I20260926 23:24:31.048377 9754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35805 every 8 connection(s)
I20260926 23:24:31.049901 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:24:31.053884 9755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:31.057816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9689
I20260926 23:24:31.058068 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:24:31.058702 388 external_mini_cluster.cc:1428] Setting key 37056e3f7db9b0e54089f17379f38016
I20260926 23:24:31.067595 9755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783: Bootstrap starting.
I20260926 23:24:31.071535 9755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:31.072770 9755 log.cc:824] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:31.073169 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:31.060637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.0.97.62:35805)
0926 23:24:31.061705 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:31.061718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:31.061764 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:24:31.063390 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:31.063416 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:31.063955 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:31.064346 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:31.065097 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:31.066389 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:31.068222 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:31.068980 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:31.069131 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:31.070485 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:31.070493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:31.070507 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:31.071067 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:31.071163 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:31.071184 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:31.071271 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:31.071640 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 23:24:31.071974 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260926 23:24:31.076330 9755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783: No bootstrap required, opened a new log
I20260926 23:24:31.084257 9755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046ecef4075a45f89a981ba9ef399783" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260926 23:24:31.084712 9755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:31.084847 9755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 046ecef4075a45f89a981ba9ef399783, State: Initialized, Role: FOLLOWER
I20260926 23:24:31.085453 9755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046ecef4075a45f89a981ba9ef399783" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260926 23:24:31.085700 9755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:24:31.085847 9755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:24:31.086045 9755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:31.088224 9755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046ecef4075a45f89a981ba9ef399783" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260926 23:24:31.088810 9755 leader_election.cc:302] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 046ecef4075a45f89a981ba9ef399783; no voters:
I20260926 23:24:31.089488 9755 leader_election.cc:288] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:24:31.089660 9760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:31.090459 9760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [term 1 LEADER]: Becoming Leader. State: Replica: 046ecef4075a45f89a981ba9ef399783, State: Running, Role: LEADER
I20260926 23:24:31.091162 9760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046ecef4075a45f89a981ba9ef399783" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } }
I20260926 23:24:31.092851 9755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:24:31.094259 9761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "046ecef4075a45f89a981ba9ef399783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046ecef4075a45f89a981ba9ef399783" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } } }
I20260926 23:24:31.094303 9762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 046ecef4075a45f89a981ba9ef399783. Latest consensus state: current_term: 1 leader_uuid: "046ecef4075a45f89a981ba9ef399783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046ecef4075a45f89a981ba9ef399783" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35805 } } }
I20260926 23:24:31.094700 9761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:31.094789 9762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783 [sys.catalog]: This master's current role is: LEADER
W20260926 23:24:31.104905 9776 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:24:31.105067 9776 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:24:31.105214 9765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:24:31.107447 9765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:24:31.116093 9765 catalog_manager.cc:1347] Generated new cluster ID: 28392041fd504888a7d0c20977d35761
I20260926 23:24:31.116252 9765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:24:31.135681 9765 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:24:31.137058 9765 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:24:31.145364 9765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 046ecef4075a45f89a981ba9ef399783: Generated new TSK 0
I20260926 23:24:31.146237 9765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:24:31.152225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--builtin_ntp_servers=127.0.97.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 23:24:31.485641 9779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:31.485939 9779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:31.486097 9779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:31.493664 9779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:31.493789 9779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:31.493983 9779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:24:31.503549 9779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:31.504971 9779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:31.506742 9779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:31.520419 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:31.520526 9784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:31.520563 9785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:31.521358 9779 server_base.cc:1034] running on GCE node
I20260926 23:24:31.522001 9779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:31.522995 9779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:31.524219 9779 hybrid_clock.cc:648] HybridClock initialized: now 1790465071524177 us; error 31 us; skew 500 ppm
I20260926 23:24:31.524573 9779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:31.532470 9779 webserver.cc:460] Webserver started at http://127.0.97.1:45379/ using document root <none> and password file <none>
I20260926 23:24:31.533295 9779 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:31.533433 9779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:31.533769 9779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:31.536664 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "794e1c1dc43d4cce8f6a4f956ac1d28d"
format_stamp: "Formatted at 2026-09-26 23:24:31 on dist-test-slave-q5g8"
server_key: "979aaa39690c3243561938cce2aaa3a7"
server_key_iv: "c6c146bc3002be0d4dd03529f1a57806"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.537523 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "794e1c1dc43d4cce8f6a4f956ac1d28d"
format_stamp: "Formatted at 2026-09-26 23:24:31 on dist-test-slave-q5g8"
server_key: "979aaa39690c3243561938cce2aaa3a7"
server_key_iv: "c6c146bc3002be0d4dd03529f1a57806"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.542629 9779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:24:31.546140 9793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:31.547489 9779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:24:31.547664 9779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "794e1c1dc43d4cce8f6a4f956ac1d28d"
format_stamp: "Formatted at 2026-09-26 23:24:31 on dist-test-slave-q5g8"
server_key: "979aaa39690c3243561938cce2aaa3a7"
server_key_iv: "c6c146bc3002be0d4dd03529f1a57806"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.547876 9779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:31.561801 9779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:31.562646 9779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:31.562920 9779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:31.563921 9779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:31.565488 9779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:31.565606 9779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:31.565735 9779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:31.565826 9779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:31.600011 9779 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38783
I20260926 23:24:31.600062 9905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38783 every 8 connection(s)
I20260926 23:24:31.601361 9779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:24:31.604655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9779
I20260926 23:24:31.604933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:24:31.605571 388 external_mini_cluster.cc:1428] Setting key bdb08013432618697c3312e6c880898d
I20260926 23:24:31.608471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--builtin_ntp_servers=127.0.97.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 23:24:31.615263 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:31.603877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51753 (local address 127.0.97.62:35805)
0926 23:24:31.604297 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:31.604303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:31.605900 (+ 1597us) server_negotiation.cc:407] Connection header received
0926 23:24:31.607882 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:31.607887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:31.607966 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:31.608084 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:31.609298 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:31.610497 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:31.612215 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:31.612992 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:31.613140 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:31.614430 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:31.614439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:31.614447 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:31.614482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:31.614545 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:31.614565 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:31.614661 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:31.614882 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 23:24:31.614992 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":206,"threads_started":1}
I20260926 23:24:31.616725 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:31.604550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.1:51753)
0926 23:24:31.605713 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:31.605760 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:31.607618 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:31.608283 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:31.608291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:31.608382 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:31.609114 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:31.609134 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:31.610647 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:31.610651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:31.612070 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:31.612087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:31.613197 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:31.613202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:31.613399 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:31.614106 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:24:31.614109 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:31.614157 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:24:31.614240 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:31.614664 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:31.614671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:31.614677 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:24:31.614837 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 23:24:31.615287 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":440,"threads_started":1}
I20260926 23:24:31.618230 9906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260926 23:24:31.618557 9906 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:31.619297 9906 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260926 23:24:31.621411 9720 ts_manager.cc:194] Registered new tserver with Master: 794e1c1dc43d4cce8f6a4f956ac1d28d (127.0.97.1:38783)
I20260926 23:24:31.622921 9720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51753
W20260926 23:24:31.933046 9910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:31.933357 9910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:31.933569 9910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:31.941962 9910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:31.942091 9910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:31.942292 9910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:24:31.952020 9910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:31.953506 9910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:31.955286 9910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:31.969301 9918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:31.969455 9915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:31.969419 9916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:31.970432 9910 server_base.cc:1034] running on GCE node
I20260926 23:24:31.971082 9910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:31.972088 9910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:31.973282 9910 hybrid_clock.cc:648] HybridClock initialized: now 1790465071973224 us; error 55 us; skew 500 ppm
I20260926 23:24:31.973609 9910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:31.975909 9910 webserver.cc:460] Webserver started at http://127.0.97.2:39051/ using document root <none> and password file <none>
I20260926 23:24:31.976639 9910 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:31.976814 9910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:31.977205 9910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:31.980216 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "fea28bd3cd8244dca0f15ef1c6518ab6"
format_stamp: "Formatted at 2026-09-26 23:24:31 on dist-test-slave-q5g8"
server_key: "5654a669a9a55250d63fb960a857023c"
server_key_iv: "fca25d2c815119332ee21691f19c8a6a"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.981087 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fea28bd3cd8244dca0f15ef1c6518ab6"
format_stamp: "Formatted at 2026-09-26 23:24:31 on dist-test-slave-q5g8"
server_key: "5654a669a9a55250d63fb960a857023c"
server_key_iv: "fca25d2c815119332ee21691f19c8a6a"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.986536 9910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 23:24:31.990218 9924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:31.991647 9910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:24:31.991791 9910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "fea28bd3cd8244dca0f15ef1c6518ab6"
format_stamp: "Formatted at 2026-09-26 23:24:31 on dist-test-slave-q5g8"
server_key: "5654a669a9a55250d63fb960a857023c"
server_key_iv: "fca25d2c815119332ee21691f19c8a6a"
server_key_version: "encryptionkey@0"
I20260926 23:24:31.992066 9910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:32.018776 9910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:32.019652 9910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:32.019937 9910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:32.020959 9910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:32.022418 9910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:32.022542 9910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:32.022684 9910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:32.022775 9910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:32.056567 9910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41437
I20260926 23:24:32.056625 10036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41437 every 8 connection(s)
I20260926 23:24:32.058103 9910 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:24:32.063477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 9910
I20260926 23:24:32.063663 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:24:32.064183 388 external_mini_cluster.cc:1428] Setting key 7c7e8c43838f787afc15934a827d2816
I20260926 23:24:32.067095 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--builtin_ntp_servers=127.0.97.20:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 23:24:32.073268 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:32.060715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47101 (local address 127.0.97.62:35805)
0926 23:24:32.060917 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:32.060923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:32.062573 (+ 1650us) server_negotiation.cc:407] Connection header received
0926 23:24:32.064934 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:32.064940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:32.065037 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:32.065156 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:32.066378 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.067484 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.070124 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.070490 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.070592 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:32.072238 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:32.072242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:32.072245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:32.072264 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:32.072355 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:32.072373 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:32.072460 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:32.072759 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 23:24:32.072903 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 23:24:32.074743 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:32.061343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.2:47101)
0926 23:24:32.062381 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:32.062429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:32.064588 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:32.065378 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:32.065386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:32.065480 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:32.066143 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:32.066160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.067641 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.067649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:32.069885 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:32.069902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.070722 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.070731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:32.071073 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:32.071865 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:24:32.071871 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:32.071927 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:24:32.072039 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:32.072479 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:32.072493 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:32.072504 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:24:32.072750 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 23:24:32.073243 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":324,"threads_started":1}
I20260926 23:24:32.076383 10037 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260926 23:24:32.076759 10037 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:32.077555 10037 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260926 23:24:32.079062 9720 ts_manager.cc:194] Registered new tserver with Master: fea28bd3cd8244dca0f15ef1c6518ab6 (127.0.97.2:41437)
I20260926 23:24:32.079922 9720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:47101
W20260926 23:24:32.401191 10040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:32.401459 10040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:32.401597 10040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:32.408957 10040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:32.409060 10040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:32.409214 10040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:24:32.418465 10040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:32.419770 10040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:32.421672 10040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:32.434516 10048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:32.434571 10046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:32.434731 10045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:32.435514 10040 server_base.cc:1034] running on GCE node
I20260926 23:24:32.436142 10040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:32.437156 10040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:32.438370 10040 hybrid_clock.cc:648] HybridClock initialized: now 1790465072438302 us; error 58 us; skew 500 ppm
I20260926 23:24:32.438688 10040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:32.440999 10040 webserver.cc:460] Webserver started at http://127.0.97.3:37907/ using document root <none> and password file <none>
I20260926 23:24:32.441699 10040 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:32.441836 10040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:32.442160 10040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:32.445052 10040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "0cf2a3ccca774666af4c07a8b2a32a0d"
format_stamp: "Formatted at 2026-09-26 23:24:32 on dist-test-slave-q5g8"
server_key: "630f5a5aaae1e1f377eb686870d78b27"
server_key_iv: "dff7f9c99421c54eac0e6061295031cb"
server_key_version: "encryptionkey@0"
I20260926 23:24:32.445858 10040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0cf2a3ccca774666af4c07a8b2a32a0d"
format_stamp: "Formatted at 2026-09-26 23:24:32 on dist-test-slave-q5g8"
server_key: "630f5a5aaae1e1f377eb686870d78b27"
server_key_iv: "dff7f9c99421c54eac0e6061295031cb"
server_key_version: "encryptionkey@0"
I20260926 23:24:32.451049 10040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:24:32.455142 10054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:32.456609 10040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:24:32.456826 10040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "0cf2a3ccca774666af4c07a8b2a32a0d"
format_stamp: "Formatted at 2026-09-26 23:24:32 on dist-test-slave-q5g8"
server_key: "630f5a5aaae1e1f377eb686870d78b27"
server_key_iv: "dff7f9c99421c54eac0e6061295031cb"
server_key_version: "encryptionkey@0"
I20260926 23:24:32.457055 10040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:32.479244 10040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:32.480135 10040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:32.480435 10040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:32.481536 10040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:32.483080 10040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:32.483203 10040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:32.483340 10040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:32.483434 10040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:32.517096 10040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44247
I20260926 23:24:32.517182 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44247 every 8 connection(s)
I20260926 23:24:32.518393 10040 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:24:32.522338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10040
I20260926 23:24:32.522544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:24:32.523068 388 external_mini_cluster.cc:1428] Setting key 4925707080cbcbd95dc142425afda10d
I20260926 23:24:32.531221 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:32.521016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34339 (local address 127.0.97.62:35805)
0926 23:24:32.521288 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:32.521292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:32.522754 (+ 1462us) server_negotiation.cc:407] Connection header received
0926 23:24:32.525232 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:32.525237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:32.525352 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:32.525460 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:32.526638 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.527350 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.528886 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.529304 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.529405 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:32.530608 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:32.530613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:32.530616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:32.530633 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:32.530660 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:32.530671 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:32.530739 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:32.530921 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 23:24:32.530991 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260926 23:24:32.532224 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:32.521537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35805 (local address 127.0.97.3:34339)
0926 23:24:32.522578 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:32.522635 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:32.524901 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:32.525644 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:32.525651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:32.525762 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:32.526494 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:32.526511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.527465 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.527469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:32.528675 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:32.528747 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.529493 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.529498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:32.529688 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:32.530319 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:24:32.530322 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:32.530358 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:24:32.530435 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:32.530751 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:32.530758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:32.530764 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:24:32.530896 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 23:24:32.531183 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":246,"threads_started":1}
I20260926 23:24:32.533555 10167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35805
I20260926 23:24:32.533828 10167 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:32.534603 10167 heartbeater.cc:511] Master 127.0.97.62:35805 requested a full tablet report, sending...
I20260926 23:24:32.535852 9720 ts_manager.cc:194] Registered new tserver with Master: 0cf2a3ccca774666af4c07a8b2a32a0d (127.0.97.3:44247)
I20260926 23:24:32.536809 9720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34339
I20260926 23:24:32.538621 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:24:32.544574 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:32.543029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.0.97.62:35805)
0926 23:24:32.543284 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:32.543290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:32.543608 (+ 318us) server_negotiation.cc:407] Connection header received
0926 23:24:32.543741 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:32.543745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:32.543826 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:32.543909 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:32.544385 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60176: BlockingRecv error: recv got EOF from 127.0.0.1:60176 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:24:32.545153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9779
I20260926 23:24:32.553048 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:32.546018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.0.97.1:38783)
0926 23:24:32.546497 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:32.546512 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:32.546537 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:24:32.546595 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:32.546601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:32.546798 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:32.546925 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:32.547237 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.548190 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.549403 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.550313 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.550467 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:32.551607 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:32.551632 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:32.551649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:32.552228 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:32.552271 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:32.552294 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:32.552382 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:32.552532 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 23:24:32.552724 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":251,"threads_started":1}
I20260926 23:24:32.554242 9901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:32.635272 9906 heartbeater.cc:503] Master 127.0.97.62:35805 was elected leader, sending a full tablet report...
I20260926 23:24:32.636850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9779
W20260926 23:24:32.652756 9705 connection.cc:535] server connection from 127.0.97.1:51753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:24:32.652933 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9910
I20260926 23:24:32.660521 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:32.653824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.97.2:41437)
0926 23:24:32.654410 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:32.654417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:32.654445 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:24:32.654505 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:32.654512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:32.654737 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:32.654864 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:32.655204 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.656184 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.657435 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.657996 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.658094 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:32.659137 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:32.659145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:32.659159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:32.659743 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:32.659788 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:32.659806 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:32.659908 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:32.660013 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 23:24:32.660147 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":298,"threads_started":1}
I20260926 23:24:32.661785 10032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:32.743112 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9910
I20260926 23:24:32.759109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10040
I20260926 23:24:32.768307 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:32.760194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.0.97.3:44247)
0926 23:24:32.761126 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:32.761136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:32.761197 (+ 61us) server_negotiation.cc:407] Connection header received
0926 23:24:32.761272 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:32.761281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:32.761644 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:32.761854 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:32.762399 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.763594 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.765045 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:32.765644 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:32.765771 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:32.766815 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:32.766824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:32.766838 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:32.767544 (+ 706us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:32.767594 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:32.767615 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:32.767710 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:32.767883 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 23:24:32.767998 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":504,"threads_started":1}
I20260926 23:24:32.769714 10162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:32.852413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10040
I20260926 23:24:32.868043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 9689
I20260926 23:24:32.869343 9750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:32.972416 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 9689
2026-09-26T23:24:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2440 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T23:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:24:33Z Disabled control of system clock
I20260926 23:24:33.046566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38877
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 23:24:33.385560 10186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:33.385880 10186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:33.386025 10186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:33.393493 10186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:24:33.393621 10186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:33.393661 10186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:33.393771 10186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:24:33.393852 10186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:24:33.403599 10186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:33.405145 10186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:33.407244 10186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:33.421849 10191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:33.421808 10194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:33.421890 10192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:33.422858 10186 server_base.cc:1034] running on GCE node
I20260926 23:24:33.423800 10186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:33.425477 10186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:33.426779 10186 hybrid_clock.cc:648] HybridClock initialized: now 1790465073426705 us; error 69 us; skew 500 ppm
I20260926 23:24:33.427199 10186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:33.429981 10186 webserver.cc:460] Webserver started at http://127.0.97.62:42535/ using document root <none> and password file <none>
I20260926 23:24:33.430734 10186 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:33.430871 10186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:33.431222 10186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:33.436035 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "2967b13bee464fd2a3296ad4591316bc"
format_stamp: "Formatted at 2026-09-26 23:24:33 on dist-test-slave-q5g8"
server_key: "256acd1b196632cc3b0c02af209128b6"
server_key_iv: "9efc788c88b1faa80cf9e8dd2590eb51"
server_key_version: "encryptionkey@0"
I20260926 23:24:33.437050 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "2967b13bee464fd2a3296ad4591316bc"
format_stamp: "Formatted at 2026-09-26 23:24:33 on dist-test-slave-q5g8"
server_key: "256acd1b196632cc3b0c02af209128b6"
server_key_iv: "9efc788c88b1faa80cf9e8dd2590eb51"
server_key_version: "encryptionkey@0"
I20260926 23:24:33.442691 10186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:24:33.446357 10200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:33.447875 10186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:24:33.448091 10186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "2967b13bee464fd2a3296ad4591316bc"
format_stamp: "Formatted at 2026-09-26 23:24:33 on dist-test-slave-q5g8"
server_key: "256acd1b196632cc3b0c02af209128b6"
server_key_iv: "9efc788c88b1faa80cf9e8dd2590eb51"
server_key_version: "encryptionkey@0"
I20260926 23:24:33.448329 10186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:33.454578 10186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:33.455399 10186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:33.455667 10186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:33.479789 10186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38877
I20260926 23:24:33.479800 10251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38877 every 8 connection(s)
I20260926 23:24:33.481278 10186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:24:33.485090 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:33.489841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10186
I20260926 23:24:33.490056 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:24:33.490873 388 external_mini_cluster.cc:1428] Setting key 0f40e731334c18e6112628850abb029c
I20260926 23:24:33.497464 10252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc: Bootstrap starting.
I20260926 23:24:33.499635 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:33.493115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.0.97.62:38877)
0926 23:24:33.494166 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:33.494179 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:33.494225 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:24:33.495943 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:33.495968 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:33.496404 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:33.496846 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:33.498021 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:33.498027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:33.498039 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:33.498052 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:33.498106 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:33.498121 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:33.498344 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:33.498478 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 23:24:33.498728 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":355,"threads_started":1}
I20260926 23:24:33.501109 10252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:33.502157 10252 log.cc:824] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:33.505172 10252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc: No bootstrap required, opened a new log
I20260926 23:24:33.512980 10252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2967b13bee464fd2a3296ad4591316bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38877 } }
I20260926 23:24:33.513423 10252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:33.513536 10252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2967b13bee464fd2a3296ad4591316bc, State: Initialized, Role: FOLLOWER
I20260926 23:24:33.514153 10252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2967b13bee464fd2a3296ad4591316bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38877 } }
I20260926 23:24:33.514427 10252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:24:33.514575 10252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:24:33.514747 10252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:33.517184 10252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2967b13bee464fd2a3296ad4591316bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38877 } }
I20260926 23:24:33.517719 10252 leader_election.cc:302] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2967b13bee464fd2a3296ad4591316bc; no voters:
I20260926 23:24:33.518455 10252 leader_election.cc:288] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:24:33.518559 10257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:33.519665 10257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [term 1 LEADER]: Becoming Leader. State: Replica: 2967b13bee464fd2a3296ad4591316bc, State: Running, Role: LEADER
I20260926 23:24:33.520406 10257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2967b13bee464fd2a3296ad4591316bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38877 } }
I20260926 23:24:33.521238 10252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:24:33.523351 10258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2967b13bee464fd2a3296ad4591316bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2967b13bee464fd2a3296ad4591316bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38877 } } }
I20260926 23:24:33.523438 10259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2967b13bee464fd2a3296ad4591316bc. Latest consensus state: current_term: 1 leader_uuid: "2967b13bee464fd2a3296ad4591316bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2967b13bee464fd2a3296ad4591316bc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38877 } } }
I20260926 23:24:33.523699 10258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:33.523808 10259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:33.525943 10263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:24:33.532472 10263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:24:33.541751 10263 catalog_manager.cc:1347] Generated new cluster ID: 64a28340962f44e3adbd239dc19122f6
I20260926 23:24:33.541887 10263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:24:33.557845 10263 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:24:33.558370 10263 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:24:33.569159 10263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2967b13bee464fd2a3296ad4591316bc: Generated new TSK 0
I20260926 23:24:33.570120 10263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:24:33.586021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38877
--builtin_ntp_servers=127.0.97.20:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 23:24:33.954526 10276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:33.954852 10276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:33.955009 10276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:33.963012 10276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:33.963164 10276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:33.963394 10276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:24:33.973069 10276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:33.974524 10276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:33.976540 10276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:33.990694 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:33.990551 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:33.991608 10284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:33.992493 10276 server_base.cc:1034] running on GCE node
I20260926 23:24:33.993175 10276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:33.994282 10276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:33.995491 10276 hybrid_clock.cc:648] HybridClock initialized: now 1790465073995420 us; error 62 us; skew 500 ppm
I20260926 23:24:33.995836 10276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:33.998366 10276 webserver.cc:460] Webserver started at http://127.0.97.1:40601/ using document root <none> and password file <none>
I20260926 23:24:33.999329 10276 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:33.999447 10276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:33.999778 10276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:34.002792 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "e5bc2dcbbcf34e5192ffb4b5a23f2f06"
format_stamp: "Formatted at 2026-09-26 23:24:33 on dist-test-slave-q5g8"
server_key: "7e486a72203536e535fafecefef1ba0a"
server_key_iv: "822faaae1b30bf702fd2abe63bf507a8"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.003685 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e5bc2dcbbcf34e5192ffb4b5a23f2f06"
format_stamp: "Formatted at 2026-09-26 23:24:33 on dist-test-slave-q5g8"
server_key: "7e486a72203536e535fafecefef1ba0a"
server_key_iv: "822faaae1b30bf702fd2abe63bf507a8"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.009014 10276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 23:24:34.012758 10290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.014326 10276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:24:34.014509 10276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "e5bc2dcbbcf34e5192ffb4b5a23f2f06"
format_stamp: "Formatted at 2026-09-26 23:24:33 on dist-test-slave-q5g8"
server_key: "7e486a72203536e535fafecefef1ba0a"
server_key_iv: "822faaae1b30bf702fd2abe63bf507a8"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.014756 10276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:34.020517 10276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:34.021461 10276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:34.021768 10276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:34.022914 10276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:34.024389 10276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:34.024497 10276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.024608 10276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:34.024746 10276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.058635 10276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46789
I20260926 23:24:34.058686 10402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46789 every 8 connection(s)
I20260926 23:24:34.060005 10276 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:24:34.065058 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10276
I20260926 23:24:34.065258 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:24:34.066071 388 external_mini_cluster.cc:1428] Setting key 546240580a1f1ccf1fd0d4e4d4db9020
I20260926 23:24:34.068784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38877
--builtin_ntp_servers=127.0.97.20:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 23:24:34.070062 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:34.062679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34557 (local address 127.0.97.62:38877)
0926 23:24:34.063126 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:34.063143 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:34.064372 (+ 1229us) server_negotiation.cc:407] Connection header received
0926 23:24:34.066839 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:34.066845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:34.066921 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:34.067038 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:34.069322 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:34.069331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:34.069339 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:34.069375 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:34.069435 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:34.069452 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:34.069558 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:34.069812 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 23:24:34.069863 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260926 23:24:34.071480 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:34.063317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38877 (local address 127.0.97.1:34557)
0926 23:24:34.064205 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:34.064244 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:34.066510 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:34.067274 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:34.067288 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:34.067391 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:34.068892 (+ 1501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:24:34.068896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:34.068956 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:24:34.069079 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:34.069569 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:34.069583 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:34.069595 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 23:24:34.069898 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 23:24:34.070216 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":267,"threads_started":1}
I20260926 23:24:34.073065 10403 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38877
I20260926 23:24:34.073374 10403 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:34.073979 10403 heartbeater.cc:511] Master 127.0.97.62:38877 requested a full tablet report, sending...
I20260926 23:24:34.076179 10217 ts_manager.cc:194] Registered new tserver with Master: e5bc2dcbbcf34e5192ffb4b5a23f2f06 (127.0.97.1:46789)
W20260926 23:24:34.419306 10407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:34.419628 10407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:34.419725 10407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:34.427316 10407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:34.427421 10407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:34.427534 10407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:24:34.436878 10407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:34.438201 10407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:34.439951 10407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:34.453959 10415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:34.454011 10413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:34.454035 10412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:34.454757 10407 server_base.cc:1034] running on GCE node
I20260926 23:24:34.455327 10407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:34.456310 10407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:34.457527 10407 hybrid_clock.cc:648] HybridClock initialized: now 1790465074457481 us; error 52 us; skew 500 ppm
I20260926 23:24:34.457841 10407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:34.460471 10407 webserver.cc:460] Webserver started at http://127.0.97.2:39513/ using document root <none> and password file <none>
I20260926 23:24:34.461279 10407 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:34.461376 10407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:34.461696 10407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:34.465071 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "fd5cbfebd8f74376bbb9adbe664c23ba"
format_stamp: "Formatted at 2026-09-26 23:24:34 on dist-test-slave-q5g8"
server_key: "636017e1aa76497ff333cbd9f6a1cf43"
server_key_iv: "ab40bec0a18f670f6dcaa0d80f2b2a5c"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.465945 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fd5cbfebd8f74376bbb9adbe664c23ba"
format_stamp: "Formatted at 2026-09-26 23:24:34 on dist-test-slave-q5g8"
server_key: "636017e1aa76497ff333cbd9f6a1cf43"
server_key_iv: "ab40bec0a18f670f6dcaa0d80f2b2a5c"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.471454 10407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:24:34.475906 10421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.477519 10407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 23:24:34.477711 10407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "fd5cbfebd8f74376bbb9adbe664c23ba"
format_stamp: "Formatted at 2026-09-26 23:24:34 on dist-test-slave-q5g8"
server_key: "636017e1aa76497ff333cbd9f6a1cf43"
server_key_iv: "ab40bec0a18f670f6dcaa0d80f2b2a5c"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.477965 10407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:34.485245 10407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:34.486131 10407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:34.486403 10407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:34.487380 10407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:34.489032 10407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:34.489118 10407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.489266 10407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:34.489351 10407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.523733 10407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43611
I20260926 23:24:34.523736 10533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43611 every 8 connection(s)
I20260926 23:24:34.525159 10407 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:24:34.533566 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:34.527836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49427 (local address 127.0.97.62:38877)
0926 23:24:34.528046 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:34.528053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:34.529520 (+ 1467us) server_negotiation.cc:407] Connection header received
0926 23:24:34.531125 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:34.531129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:34.531198 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:34.531288 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:34.532615 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:34.532622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:34.532627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:34.532653 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:34.532721 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:34.532739 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:34.532895 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:34.533171 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 23:24:34.533255 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:24:34.534113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10407
I20260926 23:24:34.534281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:24:34.534699 388 external_mini_cluster.cc:1428] Setting key 494a3dcb805c6355d919e1f3dc8be569
I20260926 23:24:34.534875 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:34.528419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38877 (local address 127.0.97.2:49427)
0926 23:24:34.529360 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:34.529401 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:34.530884 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:34.531495 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:34.531503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:34.531600 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:34.532245 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:24:34.532247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:34.532284 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:24:34.532370 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:34.532856 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:34.532868 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:34.532879 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:24:34.533232 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 23:24:34.533816 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":323,"threads_started":1}
I20260926 23:24:34.536504 10534 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38877
I20260926 23:24:34.536860 10534 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:34.537433 10534 heartbeater.cc:511] Master 127.0.97.62:38877 requested a full tablet report, sending...
I20260926 23:24:34.537434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38877
--builtin_ntp_servers=127.0.97.20:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 23:24:34.538841 10217 ts_manager.cc:194] Registered new tserver with Master: fd5cbfebd8f74376bbb9adbe664c23ba (127.0.97.2:43611)
W20260926 23:24:34.871013 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:34.871367 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:34.871537 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:34.878935 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:34.879076 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:34.879261 10537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:24:34.888633 10537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:34.889993 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:34.891884 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:34.905350 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:34.905493 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:34.905512 10545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:34.906665 10537 server_base.cc:1034] running on GCE node
I20260926 23:24:34.907404 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:34.908624 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:34.909897 10537 hybrid_clock.cc:648] HybridClock initialized: now 1790465074909835 us; error 52 us; skew 500 ppm
I20260926 23:24:34.910244 10537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:24:34.912809 10537 webserver.cc:460] Webserver started at http://127.0.97.3:38105/ using document root <none> and password file <none>
I20260926 23:24:34.913516 10537 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:34.913614 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:34.913954 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:34.917016 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed87ee8e40c24ec19c5aa5c2535b38a2"
format_stamp: "Formatted at 2026-09-26 23:24:34 on dist-test-slave-q5g8"
server_key: "664f30b686eedbfb5ab850e9e069c54a"
server_key_iv: "144f692f576b42db23d9dba19af27816"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.917785 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed87ee8e40c24ec19c5aa5c2535b38a2"
format_stamp: "Formatted at 2026-09-26 23:24:34 on dist-test-slave-q5g8"
server_key: "664f30b686eedbfb5ab850e9e069c54a"
server_key_iv: "144f692f576b42db23d9dba19af27816"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.923420 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:24:34.927340 10551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.928817 10537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 23:24:34.928994 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "ed87ee8e40c24ec19c5aa5c2535b38a2"
format_stamp: "Formatted at 2026-09-26 23:24:34 on dist-test-slave-q5g8"
server_key: "664f30b686eedbfb5ab850e9e069c54a"
server_key_iv: "144f692f576b42db23d9dba19af27816"
server_key_version: "encryptionkey@0"
I20260926 23:24:34.929291 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:34.934898 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:34.935703 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:34.935985 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:34.936951 10537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:34.938329 10537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:34.938449 10537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.938584 10537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:34.938671 10537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:34.974356 10537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42753
I20260926 23:24:34.974409 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42753 every 8 connection(s)
I20260926 23:24:34.975865 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:24:34.980051 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10537
I20260926 23:24:34.980273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:24:34.980863 388 external_mini_cluster.cc:1428] Setting key 4c651a9cacc4f1d170927ac3ca43ef60
I20260926 23:24:34.985667 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:34.978805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56533 (local address 127.0.97.62:38877)
0926 23:24:34.978992 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:34.978997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:34.980859 (+ 1862us) server_negotiation.cc:407] Connection header received
0926 23:24:34.983351 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:34.983358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:34.983439 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:34.983552 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:34.985038 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:34.985042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:34.985046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:34.985063 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:34.985100 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:34.985113 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:34.985210 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:34.985389 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 23:24:34.985438 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:24:34.986752 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:34.979397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38877 (local address 127.0.97.3:56533)
0926 23:24:34.980620 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:34.980713 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:34.983015 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:34.983776 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:34.983784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:34.983888 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:34.984718 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:24:34.984722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:34.984777 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:24:34.984881 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:34.985209 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:34.985216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:34.985221 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:24:34.985444 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 23:24:34.985675 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":354,"threads_started":1}
I20260926 23:24:34.988268 10664 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38877
I20260926 23:24:34.988590 10664 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:34.989241 10664 heartbeater.cc:511] Master 127.0.97.62:38877 requested a full tablet report, sending...
I20260926 23:24:34.990790 10217 ts_manager.cc:194] Registered new tserver with Master: ed87ee8e40c24ec19c5aa5c2535b38a2 (127.0.97.3:42753)
I20260926 23:24:34.997287 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:24:35.003561 10405 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:24:35.001979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.97.62:38877)
0926 23:24:35.002170 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:35.002174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:35.002685 (+ 511us) server_negotiation.cc:407] Connection header received
0926 23:24:35.002864 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:35.002869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:35.002954 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:35.003051 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:35.003359 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50890: BlockingRecv error: recv got EOF from 127.0.0.1:50890 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:24:35.004601 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10276
I20260926 23:24:35.008292 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:35.005577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.97.1:46789)
0926 23:24:35.006117 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:35.006125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:35.006151 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:24:35.006208 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:35.006214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:35.006429 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:35.006602 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:35.007590 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:35.007596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:35.007609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:35.007620 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:35.007660 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:35.007676 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:35.007817 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:35.007919 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 23:24:35.008028 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260926 23:24:35.009557 10398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:35.090097 10403 heartbeater.cc:503] Master 127.0.97.62:38877 was elected leader, sending a full tablet report...
I20260926 23:24:35.091568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10276
W20260926 23:24:35.106334 10201 connection.cc:535] server connection from 127.0.97.1:34557 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:24:35.106726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10407
I20260926 23:24:35.110363 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:35.107680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.0.97.2:43611)
0926 23:24:35.108252 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:35.108259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:35.108286 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:24:35.108347 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:35.108354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:35.108573 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:35.108772 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:35.109670 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:35.109676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:35.109689 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:35.109706 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:35.109743 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:35.109757 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:35.109861 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:35.110029 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 23:24:35.110108 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
I20260926 23:24:35.111565 10529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:35.192330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10407
I20260926 23:24:35.208572 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10537
I20260926 23:24:35.212867 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:35.209631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.0.97.3:42753)
0926 23:24:35.210285 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:35.210292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:35.210323 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:24:35.210384 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:35.210390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:35.210632 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:35.210816 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:35.212031 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:35.212039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:35.212051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:24:35.212068 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:35.212108 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:24:35.212123 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:35.212314 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:35.212388 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 23:24:35.212491 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":338,"threads_started":1}
I20260926 23:24:35.214183 10659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:35.295850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10537
I20260926 23:24:35.312058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10186
I20260926 23:24:35.313477 10247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:35.414198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10186
2026-09-26T23:24:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2441 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:24:35 dist-test-slave-q5g8 krb5kdc[10682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:24:35 dist-test-slave-q5g8 krb5kdc[10682](info): set up 2 sockets
Sep 26 23:24:35 dist-test-slave-q5g8 krb5kdc[10682](info): commencing operation
krb5kdc: starting...
W20260926 23:24:36.525925 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:24:36 dist-test-slave-q5g8 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:24:36.781720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40679
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:37.137298 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:37.137665 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:37.137840 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:37.145419 10695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:24:37.145582 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:37.145697 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:37.145805 10695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:24:37.145897 10695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:24:37.157722 10695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:37.159475 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:37.161118 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:37.176652 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:37.176740 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:37.176740 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:37.177508 10695 server_base.cc:1034] running on GCE node
I20260926 23:24:37.178488 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:37.180176 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:37.181438 10695 hybrid_clock.cc:648] HybridClock initialized: now 1790465077181383 us; error 58 us; skew 500 ppm
I20260926 23:24:37.181872 10695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:37 dist-test-slave-q5g8 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:37.193140 10695 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:24:37.195170 10695 webserver.cc:460] Webserver started at http://127.0.97.62:38103/ using document root <none> and password file <none>
I20260926 23:24:37.196054 10695 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:37.196245 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:37.196609 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:37.200275 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "da66a11c546e4ae891804abb671f7397"
format_stamp: "Formatted at 2026-09-26 23:24:37 on dist-test-slave-q5g8"
server_key: "ec7b7cb8563d0e5c85a79ef8bc0c2830"
server_key_iv: "a3995973ade04bcfa0dc0ec9e3cbd009"
server_key_version: "encryptionkey@0"
I20260926 23:24:37.201345 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "da66a11c546e4ae891804abb671f7397"
format_stamp: "Formatted at 2026-09-26 23:24:37 on dist-test-slave-q5g8"
server_key: "ec7b7cb8563d0e5c85a79ef8bc0c2830"
server_key_iv: "a3995973ade04bcfa0dc0ec9e3cbd009"
server_key_version: "encryptionkey@0"
I20260926 23:24:37.208381 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 23:24:37.212958 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:37.214666 10695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:24:37.214907 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "da66a11c546e4ae891804abb671f7397"
format_stamp: "Formatted at 2026-09-26 23:24:37 on dist-test-slave-q5g8"
server_key: "ec7b7cb8563d0e5c85a79ef8bc0c2830"
server_key_iv: "a3995973ade04bcfa0dc0ec9e3cbd009"
server_key_version: "encryptionkey@0"
I20260926 23:24:37.215158 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:37.231302 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:37.236861 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:37.237232 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:37.258416 10695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40679
I20260926 23:24:37.258466 10761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40679 every 8 connection(s)
I20260926 23:24:37.259908 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:24:37.264084 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:37.267562 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10695
I20260926 23:24:37.267800 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:24:37.268752 388 external_mini_cluster.cc:1428] Setting key c65156927c172476af8db4d29626021a
I20260926 23:24:37.279306 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397: Bootstrap starting.
I20260926 23:24:37.283811 10762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:37.285483 10762 log.cc:824] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:24:37 dist-test-slave-q5g8 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:37.288964 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397: No bootstrap required, opened a new log
I20260926 23:24:37.294879 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:37.271343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.0.97.62:40679)
0926 23:24:37.272296 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:37.272310 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:37.272358 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:24:37.274321 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:37.274348 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:37.274859 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:37.275272 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:37.277397 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:37.278828 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:37.280883 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:37.281643 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:37.281808 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:37.287280 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:37.287289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:37.287307 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:37.287350 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:37.290516 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:37.291292 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:37.291299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:37.291306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:37.291488 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:37.291910 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:37.291915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:37.291917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:37.292513 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:37.292896 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:37.293331 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 23:24:37.293789 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":465,"threads_started":1}
I20260926 23:24:37.298144 10762 raft_consensus.cc:348] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66a11c546e4ae891804abb671f7397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40679 } }
I20260926 23:24:37.298570 10762 raft_consensus.cc:374] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:37.298692 10762 raft_consensus.cc:729] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da66a11c546e4ae891804abb671f7397, State: Initialized, Role: FOLLOWER
I20260926 23:24:37.299358 10762 consensus_queue.cc:260] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66a11c546e4ae891804abb671f7397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40679 } }
I20260926 23:24:37.299657 10762 raft_consensus.cc:388] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:24:37.299839 10762 raft_consensus.cc:482] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:24:37.300040 10762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:37.302722 10762 raft_consensus.cc:504] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66a11c546e4ae891804abb671f7397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40679 } }
I20260926 23:24:37.303324 10762 leader_election.cc:302] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da66a11c546e4ae891804abb671f7397; no voters:
I20260926 23:24:37.304003 10762 leader_election.cc:288] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:24:37.304154 10767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:37.304973 10767 raft_consensus.cc:686] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [term 1 LEADER]: Becoming Leader. State: Replica: da66a11c546e4ae891804abb671f7397, State: Running, Role: LEADER
I20260926 23:24:37.305644 10767 consensus_queue.cc:237] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66a11c546e4ae891804abb671f7397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40679 } }
I20260926 23:24:37.306643 10762 sys_catalog.cc:564] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:24:37.309104 10769 sys_catalog.cc:455] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da66a11c546e4ae891804abb671f7397. Latest consensus state: current_term: 1 leader_uuid: "da66a11c546e4ae891804abb671f7397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66a11c546e4ae891804abb671f7397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40679 } } }
I20260926 23:24:37.309571 10769 sys_catalog.cc:458] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:37.310192 10768 sys_catalog.cc:455] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da66a11c546e4ae891804abb671f7397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66a11c546e4ae891804abb671f7397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40679 } } }
I20260926 23:24:37.310560 10768 sys_catalog.cc:458] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397 [sys.catalog]: This master's current role is: LEADER
I20260926 23:24:37.314226 10776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:24:37.321307 10776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:24:37.331064 10776 catalog_manager.cc:1347] Generated new cluster ID: 49a52e4b7d0646bf89ac4bf0e053f2b5
I20260926 23:24:37.331202 10776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:24:37.342163 10776 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:24:37.343623 10776 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:24:37.350208 10776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397: Generated new TSK 0
I20260926 23:24:37.351202 10776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:24:37.421629 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40679
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:37.778458 10788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:37.778754 10788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:37.778848 10788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:37.786223 10788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:37.786345 10788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:37.786459 10788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:24:37.796401 10788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:37.797796 10788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:37.799167 10788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:37.813233 10794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:37.813300 10796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:37.813271 10793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:37.814630 10788 server_base.cc:1034] running on GCE node
I20260926 23:24:37.815227 10788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:37.816202 10788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:37.817430 10788 hybrid_clock.cc:648] HybridClock initialized: now 1790465077817366 us; error 67 us; skew 500 ppm
I20260926 23:24:37.817759 10788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:37 dist-test-slave-q5g8 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:37.828112 10788 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:24:37.829706 10788 webserver.cc:460] Webserver started at http://127.0.97.1:35315/ using document root <none> and password file <none>
I20260926 23:24:37.830420 10788 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:37.830523 10788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:37.830777 10788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:37.833873 10788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "e08f9ceb18704f0e9d722805610787af"
format_stamp: "Formatted at 2026-09-26 23:24:37 on dist-test-slave-q5g8"
server_key: "35bebd71baeb1730beb845a5e8c2b0d9"
server_key_iv: "3b9e622bdd53030362e3f24d13f5dc91"
server_key_version: "encryptionkey@0"
I20260926 23:24:37.834585 10788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e08f9ceb18704f0e9d722805610787af"
format_stamp: "Formatted at 2026-09-26 23:24:37 on dist-test-slave-q5g8"
server_key: "35bebd71baeb1730beb845a5e8c2b0d9"
server_key_iv: "3b9e622bdd53030362e3f24d13f5dc91"
server_key_version: "encryptionkey@0"
I20260926 23:24:37.840020 10788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 23:24:37.843674 10803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:37.845360 10788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:37.845563 10788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "e08f9ceb18704f0e9d722805610787af"
format_stamp: "Formatted at 2026-09-26 23:24:37 on dist-test-slave-q5g8"
server_key: "35bebd71baeb1730beb845a5e8c2b0d9"
server_key_iv: "3b9e622bdd53030362e3f24d13f5dc91"
server_key_version: "encryptionkey@0"
I20260926 23:24:37.845845 10788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:37.866882 10788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:37.871398 10788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:37.871754 10788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:37.872830 10788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:37.874281 10788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:37.874398 10788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:37.874553 10788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:37.874626 10788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:37.908948 10788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41381
I20260926 23:24:37.908995 10915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41381 every 8 connection(s)
I20260926 23:24:37.910356 10788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:24:37.916572 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10788
I20260926 23:24:37.916821 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:24:37.917513 388 external_mini_cluster.cc:1428] Setting key 1f94975b90c13d1a94926f8fc2e89af3
Sep 26 23:24:37 dist-test-slave-q5g8 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:37.933212 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:37.913113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33151 (local address 127.0.97.62:40679)
0926 23:24:37.913545 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:37.913549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:37.914696 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 23:24:37.916846 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:37.916853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:37.916948 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:37.917099 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:37.919111 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:37.920207 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:37.922118 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:37.922826 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:37.922969 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:37.927017 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:37.927023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:37.927027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:37.927060 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:37.929673 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:37.930656 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:37.930660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:37.930663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:37.930775 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:37.931296 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:37.931299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:37.931301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:37.931559 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:37.931668 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:37.932652 (+ 984us) server_negotiation.cc:299] Negotiation successful
0926 23:24:37.932965 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260926 23:24:37.934718 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:37.913758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40679 (local address 127.0.97.1:33151)
0926 23:24:37.914517 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:37.914566 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:37.916402 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:37.917373 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:37.917382 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:37.918000 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:37.918909 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:37.918939 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:37.920390 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:37.920394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:37.921941 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:37.921955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:37.923061 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:37.923067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:37.923282 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:37.924054 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:37.924084 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:37.926740 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:37.929844 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:37.929857 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:37.929878 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:37.930527 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:37.930919 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:37.930926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:37.930933 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:37.931182 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:37.931687 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:37.931698 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:37.932193 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 23:24:37.932517 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 23:24:37.933169 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":299,"threads_started":1}
I20260926 23:24:37.936419 10916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40679
I20260926 23:24:37.936837 10916 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:37.937676 10916 heartbeater.cc:511] Master 127.0.97.62:40679 requested a full tablet report, sending...
I20260926 23:24:37.940107 10727 ts_manager.cc:194] Registered new tserver with Master: e08f9ceb18704f0e9d722805610787af (127.0.97.1:41381)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:24:37.941762 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:24:37.975754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40679
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:38.344029 10922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:38.344341 10922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:38.344436 10922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:38.351974 10922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:38.352088 10922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:38.352201 10922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:24:38.362191 10922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:38.363493 10922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:38.364866 10922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:38.378674 10927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:38.378648 10928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:38.378793 10930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:38.379491 10922 server_base.cc:1034] running on GCE node
I20260926 23:24:38.380275 10922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:38.381518 10922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:38.382745 10922 hybrid_clock.cc:648] HybridClock initialized: now 1790465078382701 us; error 42 us; skew 500 ppm
I20260926 23:24:38.383118 10922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:38 dist-test-slave-q5g8 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:38.387044 10922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:24:38.388790 10922 webserver.cc:460] Webserver started at http://127.0.97.2:41435/ using document root <none> and password file <none>
I20260926 23:24:38.389572 10922 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:38.389685 10922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:38.389969 10922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:38.393163 10922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "14f56fc371fc4246b9e9051fd91e1737"
format_stamp: "Formatted at 2026-09-26 23:24:38 on dist-test-slave-q5g8"
server_key: "c3cbad4e4a8cea65fde792eb68e4c2bd"
server_key_iv: "f5f15cb435f9fef63e9bf2245322ed72"
server_key_version: "encryptionkey@0"
I20260926 23:24:38.393941 10922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "14f56fc371fc4246b9e9051fd91e1737"
format_stamp: "Formatted at 2026-09-26 23:24:38 on dist-test-slave-q5g8"
server_key: "c3cbad4e4a8cea65fde792eb68e4c2bd"
server_key_iv: "f5f15cb435f9fef63e9bf2245322ed72"
server_key_version: "encryptionkey@0"
I20260926 23:24:38.399768 10922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 23:24:38.403590 10937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:38.405211 10922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:24:38.405412 10922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "14f56fc371fc4246b9e9051fd91e1737"
format_stamp: "Formatted at 2026-09-26 23:24:38 on dist-test-slave-q5g8"
server_key: "c3cbad4e4a8cea65fde792eb68e4c2bd"
server_key_iv: "f5f15cb435f9fef63e9bf2245322ed72"
server_key_version: "encryptionkey@0"
I20260926 23:24:38.405646 10922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:38.431713 10922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:38.436343 10922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:38.436657 10922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:38.437738 10922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:38.439147 10922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:38.439222 10922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:38.439378 10922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:38.439451 10922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:38.472563 10922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38577
I20260926 23:24:38.472651 11049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38577 every 8 connection(s)
I20260926 23:24:38.473858 10922 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:24:38.480469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 10922
I20260926 23:24:38.480654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:24:38.481130 388 external_mini_cluster.cc:1428] Setting key e9e1876460a6c04fd7cdb8c142cee897
Sep 26 23:24:38 dist-test-slave-q5g8 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:38.496215 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:38.476368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53743 (local address 127.0.97.62:40679)
0926 23:24:38.476821 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:38.476827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:38.477867 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 23:24:38.479581 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:38.479585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:38.479651 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:38.479747 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:38.481409 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:38.482207 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:38.484089 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:38.484922 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:38.485070 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:38.488842 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:38.488851 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:38.488859 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:38.488910 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:38.492805 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:38.493575 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:38.493584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:38.493590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:38.493744 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:38.494162 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:38.494170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:38.494176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:38.494595 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:38.494744 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:38.495463 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 23:24:38.495726 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":193,"threads_started":1}
I20260926 23:24:38.497066 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:38.476996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40679 (local address 127.0.97.2:53743)
0926 23:24:38.477692 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:38.477735 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:38.479334 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:38.479955 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:38.479963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:38.480469 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:38.481156 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:38.481174 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:38.482356 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:38.482361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:38.483941 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:38.483953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:38.485320 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:38.485325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:38.485533 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:38.486270 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:38.486295 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:38.488519 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:38.492958 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:38.492966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:38.492983 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:38.493445 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:38.493858 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:38.493862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:38.493864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:38.494051 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:38.494717 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:38.494723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:38.495076 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 23:24:38.495320 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 23:24:38.495805 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
I20260926 23:24:38.498562 11050 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40679
I20260926 23:24:38.498905 11050 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:38.499811 11050 heartbeater.cc:511] Master 127.0.97.62:40679 requested a full tablet report, sending...
I20260926 23:24:38.501519 10727 ts_manager.cc:194] Registered new tserver with Master: 14f56fc371fc4246b9e9051fd91e1737 (127.0.97.2:38577)
I20260926 23:24:38.502374 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53743
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:24:38.540200 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40679
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:24:38.887521 11056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:24:38.887862 11056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:24:38.888026 11056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:24:38.895692 11056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:24:38.895823 11056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:24:38.896006 11056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:24:38.906368 11056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:24:38.907804 11056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:24:38.909435 11056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:24:38.923604 11061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:38.923523 11064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:24:38.923571 11062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:24:38.926241 11056 server_base.cc:1034] running on GCE node
I20260926 23:24:38.926949 11056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:24:38.928047 11056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:24:38.929306 11056 hybrid_clock.cc:648] HybridClock initialized: now 1790465078929231 us; error 67 us; skew 500 ppm
I20260926 23:24:38.929692 11056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:24:38 dist-test-slave-q5g8 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:24:38.933795 11056 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:24:38.935449 11056 webserver.cc:460] Webserver started at http://127.0.97.3:43447/ using document root <none> and password file <none>
I20260926 23:24:38.936291 11056 fs_manager.cc:359] Metadata directory not provided
I20260926 23:24:38.936409 11056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:24:38.936759 11056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:24:38.940135 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "305d1c8647e7449a9ad0279707cfd8d4"
format_stamp: "Formatted at 2026-09-26 23:24:38 on dist-test-slave-q5g8"
server_key: "b6d541d56d6a8ceba84334028bd180e3"
server_key_iv: "757e7a553dedad8d7bbd56ad2f1d1bcb"
server_key_version: "encryptionkey@0"
I20260926 23:24:38.941059 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "305d1c8647e7449a9ad0279707cfd8d4"
format_stamp: "Formatted at 2026-09-26 23:24:38 on dist-test-slave-q5g8"
server_key: "b6d541d56d6a8ceba84334028bd180e3"
server_key_iv: "757e7a553dedad8d7bbd56ad2f1d1bcb"
server_key_version: "encryptionkey@0"
I20260926 23:24:38.945261 10916 heartbeater.cc:503] Master 127.0.97.62:40679 was elected leader, sending a full tablet report...
I20260926 23:24:38.947086 11056 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:24:38.951229 11071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:38.952836 11056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 23:24:38.953030 11056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "305d1c8647e7449a9ad0279707cfd8d4"
format_stamp: "Formatted at 2026-09-26 23:24:38 on dist-test-slave-q5g8"
server_key: "b6d541d56d6a8ceba84334028bd180e3"
server_key_iv: "757e7a553dedad8d7bbd56ad2f1d1bcb"
server_key_version: "encryptionkey@0"
I20260926 23:24:38.953323 11056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:24:38.966849 11056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:24:38.971422 11056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:24:38.971733 11056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:24:38.973093 11056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:24:38.974560 11056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:24:38.974635 11056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:38.974785 11056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:24:38.974840 11056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:24:39.011144 11056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41157
I20260926 23:24:39.011191 11183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41157 every 8 connection(s)
I20260926 23:24:39.012557 11056 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:24:39.014634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 11056
I20260926 23:24:39.014835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:24:39.015426 388 external_mini_cluster.cc:1428] Setting key 9cff6bff4740a6c182691e28a1fbaac9
Sep 26 23:24:39 dist-test-slave-q5g8 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:24:39.034992 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.015446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37353 (local address 127.0.97.62:40679)
0926 23:24:39.016007 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.016016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.017415 (+ 1399us) server_negotiation.cc:407] Connection header received
0926 23:24:39.019543 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.019548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.019623 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.019774 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:39.021531 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.022349 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.024022 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.024469 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.024569 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.028486 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:39.028492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:39.028495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:39.028529 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:39.031820 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.032739 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.032749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.032753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.032899 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.033331 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.033335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.033337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.033613 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:39.033780 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.034491 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.034720 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":269,"threads_started":1}
I20260926 23:24:39.036053 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.016180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40679 (local address 127.0.97.3:37353)
0926 23:24:39.017200 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:39.017270 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:39.019259 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:39.020041 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:39.020050 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:39.020637 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:24:39.021355 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.021375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.022493 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.022497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.023883 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.023895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.024844 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.024853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.025081 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.025858 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:24:39.025884 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:24:39.028218 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:24:39.031985 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:39.031993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:39.032011 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:39.032510 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:39.033009 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:24:39.033013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:24:39.033015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:24:39.033220 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:24:39.033773 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:24:39.033779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:24:39.034139 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 23:24:39.034382 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 23:24:39.034798 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":437,"threads_started":1}
I20260926 23:24:39.037787 11184 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40679
I20260926 23:24:39.038079 11184 heartbeater.cc:464] Registering TS with master...
I20260926 23:24:39.038827 11184 heartbeater.cc:511] Master 127.0.97.62:40679 requested a full tablet report, sending...
I20260926 23:24:39.040400 10727 ts_manager.cc:194] Registered new tserver with Master: 305d1c8647e7449a9ad0279707cfd8d4 (127.0.97.3:41157)
I20260926 23:24:39.041349 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37353
I20260926 23:24:39.044404 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:24:39.063336 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.049817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.0.97.62:40679)
0926 23:24:39.050003 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.050008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.050581 (+ 573us) server_negotiation.cc:407] Connection header received
0926 23:24:39.050859 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.050863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.050935 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.051050 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:39.052542 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.053448 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.054837 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.055261 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.055427 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.058039 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:39.058045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:39.058048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:39.058086 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:39.060566 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.061286 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.061292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.061297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.061434 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.061809 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.061815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.061818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.062144 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:39.062394 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.062739 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.062990 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:24:39.068049 10727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:24:39.075105 10727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:24:39.101225 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.087772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41381 (local address 127.0.0.1:57386)
0926 23:24:39.088775 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:39.088828 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:39.089042 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:39.090445 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:39.090453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:39.090485 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:39.090967 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.090985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.093789 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.093798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.096935 (+ 3137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.096964 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.099663 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.099674 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.100021 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.100117 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 23:24:39.100267 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 23:24:39.100569 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":832,"threads_started":1}
I20260926 23:24:39.103434 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.088094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.0.97.1:41381)
0926 23:24:39.089758 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.089767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.089796 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:24:39.089911 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.089920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.090179 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.091228 (+ 1049us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:39.091544 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.093579 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.097276 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.099334 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.102171 (+ 2837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.102564 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.102674 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.102923 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":1342,"threads_started":1}
I20260926 23:24:39.109929 10851 tablet_service.cc:1467] Processing CreateTablet for tablet f9a5339dd9e94862b47f9a4651097c26 (DEFAULT_TABLE table=test-table [id=8703280d1c53484db0f3c9ef38931756]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:39.112052 10851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a5339dd9e94862b47f9a4651097c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:39.113701 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.088315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.0.97.3:41157)
0926 23:24:39.106261 (+ 17946us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.106273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.106328 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:24:39.106471 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.106481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.106774 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.107536 (+ 762us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:39.107844 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.109133 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.111327 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.112589 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.112778 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.113028 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.113197 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.113375 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17743,"thread_start_us":17569,"threads_started":1}
I20260926 23:24:39.113721 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.088132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41157 (local address 127.0.0.1:58608)
0926 23:24:39.101618 (+ 13486us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:39.101646 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:39.101804 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:39.107015 (+ 5211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:39.107021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:39.107050 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:39.107333 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.107346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.109284 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.109314 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.111134 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.111149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.112799 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.112805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.112985 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.113024 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 23:24:39.113131 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 23:24:39.113339 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13329,"mutex_wait_us":58,"thread_start_us":18203,"threads_started":1}
I20260926 23:24:39.117033 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.090347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.0.97.2:38577)
0926 23:24:39.104374 (+ 14027us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.104389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.104437 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:24:39.104581 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.104596 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.105039 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.105353 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:39.107093 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.109126 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.112889 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.114505 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.114625 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.114858 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.116424 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.116618 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13797,"thread_start_us":13651,"threads_started":1}
I20260926 23:24:39.117475 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.089789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38577 (local address 127.0.0.1:42946)
0926 23:24:39.103824 (+ 14035us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:39.103852 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:39.104020 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:39.105329 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:39.105335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:39.105371 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:39.105865 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.105883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.109374 (+ 3491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.109382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.112622 (+ 3240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.112640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.115356 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.115369 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.115608 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.115651 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:24:39.115764 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 23:24:39.115920 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13885,"mutex_wait_us":129,"thread_start_us":13691,"threads_started":1}
I20260926 23:24:39.118147 11119 tablet_service.cc:1467] Processing CreateTablet for tablet f9a5339dd9e94862b47f9a4651097c26 (DEFAULT_TABLE table=test-table [id=8703280d1c53484db0f3c9ef38931756]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:39.119683 11119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a5339dd9e94862b47f9a4651097c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:39.121848 10985 tablet_service.cc:1467] Processing CreateTablet for tablet f9a5339dd9e94862b47f9a4651097c26 (DEFAULT_TABLE table=test-table [id=8703280d1c53484db0f3c9ef38931756]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:24:39.123456 10985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a5339dd9e94862b47f9a4651097c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:24:39.129559 11203 tablet_bootstrap.cc:492] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Bootstrap starting.
I20260926 23:24:39.133292 11203 tablet_bootstrap.cc:654] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:39.134485 11203 log.cc:824] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:39.138434 11204 tablet_bootstrap.cc:492] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Bootstrap starting.
I20260926 23:24:39.139564 11206 tablet_bootstrap.cc:492] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Bootstrap starting.
I20260926 23:24:39.139704 11203 tablet_bootstrap.cc:492] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: No bootstrap required, opened a new log
I20260926 23:24:39.140496 11203 ts_tablet_manager.cc:1397] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260926 23:24:39.142556 11206 tablet_bootstrap.cc:654] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:39.142601 11204 tablet_bootstrap.cc:654] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Neither blocks nor log segments found. Creating new log.
I20260926 23:24:39.143646 11206 log.cc:824] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:39.143956 11204 log.cc:824] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Log is configured to *not* fsync() on all Append() calls
I20260926 23:24:39.148496 11206 tablet_bootstrap.cc:492] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: No bootstrap required, opened a new log
I20260926 23:24:39.148852 11206 ts_tablet_manager.cc:1397] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 23:24:39.149529 11204 tablet_bootstrap.cc:492] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: No bootstrap required, opened a new log
I20260926 23:24:39.149854 11204 ts_tablet_manager.cc:1397] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260926 23:24:39.152626 11203 raft_consensus.cc:348] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.153208 11203 raft_consensus.cc:374] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:39.153324 11203 raft_consensus.cc:729] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08f9ceb18704f0e9d722805610787af, State: Initialized, Role: FOLLOWER
I20260926 23:24:39.154079 11203 consensus_queue.cc:260] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.156222 11203 ts_tablet_manager.cc:1428] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260926 23:24:39.157642 11206 raft_consensus.cc:348] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.158133 11206 raft_consensus.cc:374] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:39.158258 11206 raft_consensus.cc:729] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f56fc371fc4246b9e9051fd91e1737, State: Initialized, Role: FOLLOWER
I20260926 23:24:39.158890 11206 consensus_queue.cc:260] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.159952 11050 heartbeater.cc:503] Master 127.0.97.62:40679 was elected leader, sending a full tablet report...
I20260926 23:24:39.161526 11204 raft_consensus.cc:348] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.161926 11206 ts_tablet_manager.cc:1428] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260926 23:24:39.162057 11204 raft_consensus.cc:374] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:24:39.162201 11204 raft_consensus.cc:729] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305d1c8647e7449a9ad0279707cfd8d4, State: Initialized, Role: FOLLOWER
W20260926 23:24:39.162771 10917 tablet.cc:2369] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:24:39.162931 11204 consensus_queue.cc:260] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.164213 11184 heartbeater.cc:503] Master 127.0.97.62:40679 was elected leader, sending a full tablet report...
I20260926 23:24:39.165156 11204 ts_tablet_manager.cc:1428] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260926 23:24:39.225873 11051 tablet.cc:2369] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:24:39.264330 11185 tablet.cc:2369] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:24:39.388484 11209 raft_consensus.cc:482] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:24:39.388978 11209 raft_consensus.cc:504] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.390830 11209 leader_election.cc:288] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305d1c8647e7449a9ad0279707cfd8d4 (127.0.97.3:41157), 14f56fc371fc4246b9e9051fd91e1737 (127.0.97.2:38577)
I20260926 23:24:39.398545 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.391392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38397 (local address 127.0.97.2:38577)
0926 23:24:39.391647 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.391657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.392541 (+ 884us) server_negotiation.cc:407] Connection header received
0926 23:24:39.392777 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.392785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.392851 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.392960 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:39.393726 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.394471 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.396503 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.397423 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.397590 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.397706 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.398107 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.398244 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:24:39.398545 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.391462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54051 (local address 127.0.97.3:41157)
0926 23:24:39.391741 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.391750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.392489 (+ 739us) server_negotiation.cc:407] Connection header received
0926 23:24:39.392753 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.392757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.392823 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.392935 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:24:39.393585 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.394411 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.396547 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.397460 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.397620 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.397743 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.398110 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.398250 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:24:39.398697 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.391090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41157 (local address 127.0.97.1:54051)
0926 23:24:39.392339 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:39.392358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:39.392566 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:39.392963 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:39.392967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:39.392994 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:39.393421 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.393430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.394558 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.394566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.396370 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.396383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.397729 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.397736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.397943 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.397969 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 23:24:39.398082 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 23:24:39.398259 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":124,"thread_start_us":948,"threads_started":1}
I20260926 23:24:39.399186 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.391265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38577 (local address 127.0.97.1:38397)
0926 23:24:39.392408 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0926 23:24:39.392424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:24:39.392603 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:24:39.392978 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:24:39.392982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:24:39.393006 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:24:39.393536 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.393546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.394634 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.394640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.396324 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:24:39.396337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.397682 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.397689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:24:39.397943 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.397974 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:24:39.398080 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:24:39.398246 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":24,"thread_start_us":637,"threads_started":1}
I20260926 23:24:39.399973 11139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5339dd9e94862b47f9a4651097c26" candidate_uuid: "e08f9ceb18704f0e9d722805610787af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305d1c8647e7449a9ad0279707cfd8d4" is_pre_election: true
I20260926 23:24:39.400488 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5339dd9e94862b47f9a4651097c26" candidate_uuid: "e08f9ceb18704f0e9d722805610787af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f56fc371fc4246b9e9051fd91e1737" is_pre_election: true
I20260926 23:24:39.400597 11139 raft_consensus.cc:2454] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08f9ceb18704f0e9d722805610787af in term 0.
I20260926 23:24:39.401062 11005 raft_consensus.cc:2454] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08f9ceb18704f0e9d722805610787af in term 0.
I20260926 23:24:39.401365 10807 leader_election.cc:302] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305d1c8647e7449a9ad0279707cfd8d4, e08f9ceb18704f0e9d722805610787af; no voters:
I20260926 23:24:39.401783 11209 raft_consensus.cc:2781] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:24:39.401934 11209 raft_consensus.cc:482] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:24:39.402056 11209 raft_consensus.cc:3037] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:39.404265 11209 raft_consensus.cc:504] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.405028 11209 leader_election.cc:288] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [CANDIDATE]: Term 1 election: Requested vote from peers 305d1c8647e7449a9ad0279707cfd8d4 (127.0.97.3:41157), 14f56fc371fc4246b9e9051fd91e1737 (127.0.97.2:38577)
I20260926 23:24:39.405467 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5339dd9e94862b47f9a4651097c26" candidate_uuid: "e08f9ceb18704f0e9d722805610787af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f56fc371fc4246b9e9051fd91e1737"
I20260926 23:24:39.405467 11139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5339dd9e94862b47f9a4651097c26" candidate_uuid: "e08f9ceb18704f0e9d722805610787af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305d1c8647e7449a9ad0279707cfd8d4"
I20260926 23:24:39.405680 11139 raft_consensus.cc:3037] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:39.405680 11005 raft_consensus.cc:3037] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:24:39.408929 11139 raft_consensus.cc:2454] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08f9ceb18704f0e9d722805610787af in term 1.
I20260926 23:24:39.408933 11005 raft_consensus.cc:2454] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08f9ceb18704f0e9d722805610787af in term 1.
I20260926 23:24:39.409360 10806 leader_election.cc:302] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f56fc371fc4246b9e9051fd91e1737, e08f9ceb18704f0e9d722805610787af; no voters:
I20260926 23:24:39.409699 11209 raft_consensus.cc:2781] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:24:39.410362 11209 raft_consensus.cc:686] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 1 LEADER]: Becoming Leader. State: Replica: e08f9ceb18704f0e9d722805610787af, State: Running, Role: LEADER
I20260926 23:24:39.410912 11209 consensus_queue.cc:237] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } }
I20260926 23:24:39.415895 10727 catalog_manager.cc:5495] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af reported cstate change: term changed from 0 to 1, leader changed from <none> to e08f9ceb18704f0e9d722805610787af (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e08f9ceb18704f0e9d722805610787af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f9ceb18704f0e9d722805610787af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 } health_report { overall_health: UNKNOWN } } }
I20260926 23:24:39.486490 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.480067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.0.97.1:41381)
0926 23:24:39.480262 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.480271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.480296 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:24:39.480565 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.480569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.480654 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.480826 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:24:39.481283 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.482035 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.484094 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.484530 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.484637 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.485293 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:24:39.485857 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:24:39.485975 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.486122 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.486197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:24:39.496101 11005 raft_consensus.cc:1261] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 1 FOLLOWER]: Refusing update from remote peer e08f9ceb18704f0e9d722805610787af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:39.496109 11139 raft_consensus.cc:1261] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 1 FOLLOWER]: Refusing update from remote peer e08f9ceb18704f0e9d722805610787af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:24:39.497280 11214 consensus_queue.cc:1035] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f56fc371fc4246b9e9051fd91e1737" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:39.497714 11209 consensus_queue.cc:1035] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [LEADER]: Connected to new peer: Peer: permanent_uuid: "305d1c8647e7449a9ad0279707cfd8d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:24:39.524904 10726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:39.525209 10726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:24:39.528884 10726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da66a11c546e4ae891804abb671f7397: Sending DeleteTablet for 3 replicas of tablet f9a5339dd9e94862b47f9a4651097c26
I20260926 23:24:39.530256 10851 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a5339dd9e94862b47f9a4651097c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57386
I20260926 23:24:39.530249 11119 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a5339dd9e94862b47f9a4651097c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58608
I20260926 23:24:39.530335 10985 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a5339dd9e94862b47f9a4651097c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:24:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42946
I20260926 23:24:39.532928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10788
I20260926 23:24:39.542484 11227 tablet_replica.cc:330] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: stopping tablet replica
I20260926 23:24:39.543339 11227 raft_consensus.cc:2229] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:24:39.544452 11227 raft_consensus.cc:2258] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:39.545593 11226 tablet_replica.cc:330] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: stopping tablet replica
I20260926 23:24:39.548374 11227 ts_tablet_manager.cc:1905] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:39.550596 11226 raft_consensus.cc:2229] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:39.551283 11226 raft_consensus.cc:2258] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:39.554952 11226 ts_tablet_manager.cc:1905] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:24:39.556190 11228 tablet_replica.cc:330] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: stopping tablet replica
I20260926 23:24:39.556705 11228 raft_consensus.cc:2229] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:24:39.557431 11228 raft_consensus.cc:2258] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:24:39.560417 11228 ts_tablet_manager.cc:1905] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:24:39 dist-test-slave-q5g8 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:24:39.561857 11227 ts_tablet_manager.cc:1918] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:39.562042 11227 log.cc:1196] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/f9a5339dd9e94862b47f9a4651097c26
I20260926 23:24:39.562608 11227 ts_tablet_manager.cc:1939] T f9a5339dd9e94862b47f9a4651097c26 P e08f9ceb18704f0e9d722805610787af: Deleting consensus metadata
I20260926 23:24:39.563347 11226 ts_tablet_manager.cc:1918] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:39.563607 11226 log.cc:1196] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/f9a5339dd9e94862b47f9a4651097c26
I20260926 23:24:39.564211 11226 ts_tablet_manager.cc:1939] T f9a5339dd9e94862b47f9a4651097c26 P 305d1c8647e7449a9ad0279707cfd8d4: Deleting consensus metadata
I20260926 23:24:39.564824 10713 catalog_manager.cc:4841] TS e08f9ceb18704f0e9d722805610787af (127.0.97.1:41381): tablet f9a5339dd9e94862b47f9a4651097c26 (table test-table [id=8703280d1c53484db0f3c9ef38931756]) successfully deleted
I20260926 23:24:39.566998 10714 catalog_manager.cc:4841] TS 305d1c8647e7449a9ad0279707cfd8d4 (127.0.97.3:41157): tablet f9a5339dd9e94862b47f9a4651097c26 (table test-table [id=8703280d1c53484db0f3c9ef38931756]) successfully deleted
I20260926 23:24:39.568163 11228 ts_tablet_manager.cc:1918] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:24:39.568377 11228 log.cc:1196] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/f9a5339dd9e94862b47f9a4651097c26
I20260926 23:24:39.569000 11228 ts_tablet_manager.cc:1939] T f9a5339dd9e94862b47f9a4651097c26 P 14f56fc371fc4246b9e9051fd91e1737: Deleting consensus metadata
I20260926 23:24:39.570502 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.534906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.0.97.1:41381)
0926 23:24:39.535123 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.535132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.549417 (+ 14285us) server_negotiation.cc:407] Connection header received
0926 23:24:39.550000 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.550006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.550122 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.550206 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:39.552183 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.553492 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.555545 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.556799 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.556930 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.562511 (+ 5581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:39.562525 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:39.562544 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:39.562606 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:39.566838 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.567835 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.567857 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.567863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.568062 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.568616 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.568625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.568647 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.569171 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:39.569472 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.569829 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.570107 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:24:39.571128 10713 catalog_manager.cc:4841] TS 14f56fc371fc4246b9e9051fd91e1737 (127.0.97.2:38577): tablet f9a5339dd9e94862b47f9a4651097c26 (table test-table [id=8703280d1c53484db0f3c9ef38931756]) successfully deleted
I20260926 23:24:39.571257 10911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:39.673309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10788
I20260926 23:24:39.693732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10922
Sep 26 23:24:39 dist-test-slave-q5g8 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:24:39.708928 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.694618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.0.97.2:38577)
0926 23:24:39.694804 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.694810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.694870 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:24:39.695090 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.695096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.695175 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.695299 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:39.696566 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.697374 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.698651 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.699073 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.699211 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.703248 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:39.703264 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:39.703281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:39.703316 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:39.705985 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.706750 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.706765 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.706767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.706909 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.707290 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.707296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.707299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.707738 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:39.708043 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.708338 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.708552 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:24:39.709645 11045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:39.801532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10922
I20260926 23:24:39.821781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 11056
Sep 26 23:24:39 dist-test-slave-q5g8 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:24:39.839339 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:24:39.822662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.0.97.3:41157)
0926 23:24:39.822886 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:24:39.822893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:24:39.822918 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:24:39.823144 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:24:39.823151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:24:39.823238 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:24:39.823368 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:24:39.824904 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.825696 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.827219 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:24:39.827734 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:24:39.827908 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:24:39.833081 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:24:39.833101 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:24:39.833120 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:24:39.833189 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:24:39.836171 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.837110 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.837118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.837121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.837276 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:24:39.837715 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:24:39.837719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:24:39.837721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:24:39.838101 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:24:39.838423 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 23:24:39.838753 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 23:24:39.838977 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:24:39.840063 11179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:39.929421 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 11056
I20260926 23:24:39.947072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 10695
I20260926 23:24:39.948222 10757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:24:40.058651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 10695
2026-09-26T23:24:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4646 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:24:40 dist-test-slave-q5g8 krb5kdc[11236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:24:40 dist-test-slave-q5g8 krb5kdc[11236](info): set up 2 sockets
Sep 26 23:24:40 dist-test-slave-q5g8 krb5kdc[11236](info): commencing operation
krb5kdc: starting...
W20260926 23:24:41.185768 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:24:41 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:24:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 23:24:41.547610 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknM1jIl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/postgres -l logfile start
2026-09-26 23:24:44.390 UTC [11266] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 23:24:44.390 UTC [11266] LOG: listening on IPv4 address "127.0.97.20", port 57265
2026-09-26 23:24:44.394 UTC [11266] LOG: listening on Unix socket "/tmp/.s.PGSQL.57265"
2026-09-26 23:24:44.419 UTC [11269] LOG: database system was shut down at 2026-09-26 23:24:42 UTC
2026-09-26 23:24:44.424 UTC [11266] LOG: database system is ready to accept connections
I20260926 23:24:45.405315 388 mini_postgres.cc:96] Postgres bound to 57265
2026-09-26 23:24:45.423 UTC [11277] FATAL: database "slave" does not exist
127.0.97.20:57265 - accepting connections
I20260926 23:24:45.424335 388 mini_ranger.cc:161] Starting Ranger...
I20260926 23:24:45.453495 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 23:24:46.011886 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 23:24:46.012058 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-admin
2026-09-26 23:24:46,329 [I] DB FLAVOR :POSTGRES
2026-09-26 23:24:46,329 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:24:46,329 [I] Checking connection..
2026-09-26 23:24:46,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 23:24:46,584 [I] Checking connection passed.
2026-09-26 23:24:46,584 [I] --------- Verifying version history table ---------
2026-09-26 23:24:46,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:24:46,867 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:24:46,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:24:47,144 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:24:47,144 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 23:24:47,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 23:24:47.424 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 23:24:47,431 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 23:24:47,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:24:47,683 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 23:24:47,684 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:24:47,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 23:24:47,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 23:24:48,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q5g8.c.gcp-upstream.internal','N') ;"
2026-09-26 23:24:48,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:24:48,746 [I] Table x_portal_user does not exist in database ranger
2026-09-26 23:24:48,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:24:49,016 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 23:24:49,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:24:49,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:24:49,526 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 23:24:49,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 23:24:50.586 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.604 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.623 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.641 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.659 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.678 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.738 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.747 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.764 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.776 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.788 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.803 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.814 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.822 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.830 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:50.838 UTC [11449] WARNING: there is no transaction in progress
2026-09-26 23:24:51,581 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 23:24:51,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 23:24:51,842 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 23:24:51,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:24:52,136 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 23:24:52,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:24:52,423 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 23:24:52,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:24:52,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:24:52,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q5g8.c.gcp-upstream.internal';"
2026-09-26 23:24:53,175 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 23:24:53,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 23:24:53,430 [I] DB_PATCHES have already been applied
I20260926 23:24:53.436213 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknM1jIl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 23:24:53.436333 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 23:24:53.452498 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:57999
Sep 26, 2026 11:24:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:24:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:24:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:24:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:24:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:24:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:24:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 23:24:54.623695 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.199s user 0.011s sys 0.031s
Sep 26, 2026 11:25:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:25:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 11:25:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:25:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:25:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 23:25:17.709686 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 23:25:17 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465117, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 23:25:17.910534 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 23:25:17.937737 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 23:25:18.990765 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 23:25:18.990931 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms
I20260926 23:25:18.991080 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:57265
2026-09-26 23:25:19,429 [I] DB FLAVOR :POSTGRES
2026-09-26 23:25:19,430 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:25:19,430 [I] Checking connection
2026-09-26 23:25:19,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 23:25:19,702 [I] connection success
2026-09-26 23:25:19,703 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 23:25:19,703 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 23:25:19,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 23:25:19,960 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 23:25:19,961 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:25:19,961 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 23:25:19,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 23:25:20,279 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 23:25:20.853662 388 mini_ranger_kms.cc:310] Created kms service
I20260926 23:25:21.063480 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 23:25:21.118134 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 23:25:21 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465117, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:25:21.557255 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 23:25:21.557588 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 23:25:21.557649 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 23:25:21.583906 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11630
I20260926 23:25:21.584131 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:42989
Sep 26, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:25:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:25:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 23:25:24.010007 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.100s user 0.127s sys 0.337s
I20260926 23:25:24.010280 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 23:25:24.010511 388 mini_ranger_kms.cc:401] 127.0.97.20:42989/kms/v1/keys
Sep 26 23:25:24 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 23:25:24 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:25:25 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:25:27 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:25:27.600518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43591
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43591
--ranger_config_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42989/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:25:27.970327 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:25:27.970656 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:25:27.970798 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:25:27.978920 11680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:25:27.979058 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:25:27.979171 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:25:27.979266 11680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:25:27.979314 11680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:25:27.990264 11680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:42989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43591
--ranger_config_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:25:27.991751 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:25:27.993260 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:25:28.006651 11686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:28.006749 11685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:28.006772 11688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:25:28.007936 11680 server_base.cc:1034] running on GCE node
I20260926 23:25:28.008932 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:25:28.010565 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:25:28.011788 11680 hybrid_clock.cc:648] HybridClock initialized: now 1790465128011743 us; error 35 us; skew 500 ppm
I20260926 23:25:28.012219 11680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:25:28 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:25:28.023386 11680 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:25:28.025277 11680 webserver.cc:460] Webserver started at http://127.0.97.62:36479/ using document root <none> and password file <none>
I20260926 23:25:28.026067 11680 fs_manager.cc:359] Metadata directory not provided
I20260926 23:25:28.026221 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:25:28.026566 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:25:28 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:25:28.098348 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "0d6b7f15dac7492281aa76660ce288ff"
format_stamp: "Formatted at 2026-09-26 23:25:28 on dist-test-slave-q5g8"
server_key: "ed7cf87970225436051d185e800d0231"
server_key_iv: "e0c468ba59722e536c72210285fce217"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:28.099640 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d6b7f15dac7492281aa76660ce288ff"
format_stamp: "Formatted at 2026-09-26 23:25:28 on dist-test-slave-q5g8"
server_key: "ed7cf87970225436051d185e800d0231"
server_key_iv: "e0c468ba59722e536c72210285fce217"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:28.105611 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:25:28.123518 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:28.125633 11680 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:25:28.125869 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "0d6b7f15dac7492281aa76660ce288ff"
format_stamp: "Formatted at 2026-09-26 23:25:28 on dist-test-slave-q5g8"
server_key: "ed7cf87970225436051d185e800d0231"
server_key_iv: "e0c468ba59722e536c72210285fce217"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:28.126117 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:25:28.140486 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:25:28.145669 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:25:28.146044 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:25:28.167735 11680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43591
I20260926 23:25:28.167759 11746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43591 every 8 connection(s)
I20260926 23:25:28.169433 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:25:28.174203 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:25:28.179572 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 11680
I20260926 23:25:28.179802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:25:28.180626 388 external_mini_cluster.cc:1428] Setting key c756d2535a087e1c2f373274aa27281b
I20260926 23:25:28.187409 11747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff: Bootstrap starting.
I20260926 23:25:28.190884 11747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff: Neither blocks nor log segments found. Creating new log.
I20260926 23:25:28.192065 11747 log.cc:824] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff: Log is configured to *not* fsync() on all Append() calls
I20260926 23:25:28.195365 11747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff: No bootstrap required, opened a new log
Sep 26 23:25:28 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:25:28.204866 11747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6b7f15dac7492281aa76660ce288ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43591 } }
I20260926 23:25:28.205281 11747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:25:28.205420 11747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6b7f15dac7492281aa76660ce288ff, State: Initialized, Role: FOLLOWER
I20260926 23:25:28.205912 11750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:28.182909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.0.97.62:43591)
0926 23:25:28.183845 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:28.183860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:28.183908 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:25:28.185821 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:28.185849 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:28.186303 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:28.186745 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:28.188579 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:28.190059 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:28.191506 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:28.192348 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:28.192593 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:28.196952 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:28.196962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:28.196980 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:28.197036 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:28.201104 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:28.202016 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:28.202026 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:28.202034 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:28.202216 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:28.202653 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:28.202662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:28.202672 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:28.203339 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:28.203713 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:28.204167 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 23:25:28.204622 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":395,"threads_started":1}
I20260926 23:25:28.206271 11747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6b7f15dac7492281aa76660ce288ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43591 } }
I20260926 23:25:28.206588 11747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:25:28.206794 11747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:25:28.207000 11747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:25:28.209798 11747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6b7f15dac7492281aa76660ce288ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43591 } }
I20260926 23:25:28.210407 11747 leader_election.cc:302] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d6b7f15dac7492281aa76660ce288ff; no voters:
I20260926 23:25:28.211107 11747 leader_election.cc:288] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:25:28.211278 11752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:25:28.212173 11752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [term 1 LEADER]: Becoming Leader. State: Replica: 0d6b7f15dac7492281aa76660ce288ff, State: Running, Role: LEADER
I20260926 23:25:28.212857 11752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6b7f15dac7492281aa76660ce288ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43591 } }
I20260926 23:25:28.213614 11747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:25:28.216671 11754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d6b7f15dac7492281aa76660ce288ff. Latest consensus state: current_term: 1 leader_uuid: "0d6b7f15dac7492281aa76660ce288ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6b7f15dac7492281aa76660ce288ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43591 } } }
I20260926 23:25:28.217298 11754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [sys.catalog]: This master's current role is: LEADER
I20260926 23:25:28.216781 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d6b7f15dac7492281aa76660ce288ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6b7f15dac7492281aa76660ce288ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43591 } } }
I20260926 23:25:28.217427 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff [sys.catalog]: This master's current role is: LEADER
I20260926 23:25:28.221819 11758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:25:28.230010 11758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:25:28.237577 11747 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 23:25:28.251173 11758 catalog_manager.cc:1347] Generated new cluster ID: 4d7cc96cfae941fc963933fcef2e6538
I20260926 23:25:28.251482 11758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:25:28.268980 11758 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:25:28.270488 11758 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:25:28.276858 11758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff: Generated new TSK 0
I20260926 23:25:28.277875 11758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:25:29.174990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43591
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:25:29 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 23:25:29.646534 11792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:25:29.647042 11792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:25:29.647226 11792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:25:29.658483 11792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:25:29.658643 11792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:25:29.658900 11792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:25:29.676057 11792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:42989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:25:29.677986 11792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:25:29.679917 11792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:25:29.701180 11802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:29.702497 11803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:29.714571 11805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:25:29.715509 11792 server_base.cc:1034] running on GCE node
I20260926 23:25:29.716475 11792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:25:29.718003 11792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:25:29.719348 11792 hybrid_clock.cc:648] HybridClock initialized: now 1790465129719273 us; error 63 us; skew 500 ppm
I20260926 23:25:29.719797 11792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:25:29 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:25:29.725631 11792 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:25:29.731650 11792 webserver.cc:460] Webserver started at http://127.0.97.1:46211/ using document root <none> and password file <none>
I20260926 23:25:29.733186 11792 fs_manager.cc:359] Metadata directory not provided
I20260926 23:25:29.733457 11792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:25:29.734005 11792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:25:29 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:25:29.762387 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "f0c96efd3a0d4556a6e4f432f4e99076"
format_stamp: "Formatted at 2026-09-26 23:25:29 on dist-test-slave-q5g8"
server_key: "fb7ce75dc564773bb873d9f4b61876fd"
server_key_iv: "5c1b5f0104f22908b0fb501fb0ef069e"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:29.764155 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f0c96efd3a0d4556a6e4f432f4e99076"
format_stamp: "Formatted at 2026-09-26 23:25:29 on dist-test-slave-q5g8"
server_key: "fb7ce75dc564773bb873d9f4b61876fd"
server_key_iv: "5c1b5f0104f22908b0fb501fb0ef069e"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:29.787501 11792 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.006s sys 0.013s
I20260926 23:25:29.809343 11812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:29.811208 11792 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260926 23:25:29.811441 11792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "f0c96efd3a0d4556a6e4f432f4e99076"
format_stamp: "Formatted at 2026-09-26 23:25:29 on dist-test-slave-q5g8"
server_key: "fb7ce75dc564773bb873d9f4b61876fd"
server_key_iv: "5c1b5f0104f22908b0fb501fb0ef069e"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:29.811728 11792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:25:29.842906 11792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:25:29.848922 11792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:25:29.849344 11792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:25:29.853133 11792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:25:29.858186 11792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:25:29.858371 11792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:25:29.858553 11792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:25:29.858659 11792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:29.965662 11792 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44205
I20260926 23:25:29.965713 11924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44205 every 8 connection(s)
I20260926 23:25:29.967762 11792 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:25:29.970118 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 11792
I20260926 23:25:29.970325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:25:29.971666 388 external_mini_cluster.cc:1428] Setting key d156cd77ef4e5d119259f3de9c325cd7
Sep 26 23:25:30 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:25:30.017198 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:29.978754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50999 (local address 127.0.97.62:43591)
0926 23:25:29.985537 (+ 6783us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:29.985545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:29.990412 (+ 4867us) server_negotiation.cc:407] Connection header received
0926 23:25:29.990515 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:29.990521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:29.990640 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:29.990773 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:29.993767 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:29.994982 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:29.997715 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:29.998355 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:29.998503 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:30.006217 (+ 7714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:30.006223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:30.006227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:30.006269 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:30.010179 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:30.011972 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:30.011980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:30.011985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:30.012234 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:30.013157 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:30.013163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:30.013166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:30.013574 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:30.013742 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:30.016130 (+ 2388us) server_negotiation.cc:299] Negotiation successful
0926 23:25:30.016752 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6606,"thread_start_us":6476,"threads_started":1}
I20260926 23:25:30.018982 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:29.978077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43591 (local address 127.0.97.1:50999)
0926 23:25:29.986584 (+ 8507us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:29.986671 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:29.989776 (+ 3105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:29.991319 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:29.991334 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:29.992511 (+ 1177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:25:29.993501 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:29.993533 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:29.995210 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:29.995215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:29.997478 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:29.997511 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:29.998875 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:29.998882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:29.999190 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:30.000267 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:25:30.000304 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:25:30.005845 (+ 5541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:25:30.010907 (+ 5062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:25:30.010921 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:25:30.010946 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:25:30.011734 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:25:30.012546 (+ 812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:25:30.012554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:25:30.012558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:25:30.012928 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:25:30.013832 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:25:30.013842 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:25:30.014397 (+ 555us) client_negotiation.cc:769] Sending connection context
0926 23:25:30.014776 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 23:25:30.015467 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8084,"thread_start_us":7843,"threads_started":1}
I20260926 23:25:30.023195 11925 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43591
I20260926 23:25:30.023634 11925 heartbeater.cc:464] Registering TS with master...
I20260926 23:25:30.024669 11925 heartbeater.cc:511] Master 127.0.97.62:43591 requested a full tablet report, sending...
I20260926 23:25:30.028296 11712 ts_manager.cc:194] Registered new tserver with Master: f0c96efd3a0d4556a6e4f432f4e99076 (127.0.97.1:44205)
I20260926 23:25:30.030783 11712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50999
Sep 26 23:25:30 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:25:30.068347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43591
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:25:30 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 23:25:30.627749 11931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:25:30.628141 11931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:25:30.628351 11931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:25:30.638887 11931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:25:30.639074 11931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:25:30.639285 11931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:25:30.655323 11931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:42989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:25:30.657609 11931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:25:30.659807 11931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:25:30.706426 11938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:30.707229 11939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:30.710865 11942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:25:30.713091 11931 server_base.cc:1034] running on GCE node
I20260926 23:25:30.713992 11931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:25:30.715317 11931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:25:30.716636 11931 hybrid_clock.cc:648] HybridClock initialized: now 1790465130716541 us; error 87 us; skew 500 ppm
I20260926 23:25:30.717142 11931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:25:30 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:25:30.720865 11931 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:25:30.722939 11931 webserver.cc:460] Webserver started at http://127.0.97.2:36547/ using document root <none> and password file <none>
I20260926 23:25:30.723783 11931 fs_manager.cc:359] Metadata directory not provided
I20260926 23:25:30.723953 11931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:25:30.724318 11931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:25:30 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:25:30.742259 11931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "df99faf97aa74d04b7988b7e475ae24c"
format_stamp: "Formatted at 2026-09-26 23:25:30 on dist-test-slave-q5g8"
server_key: "e3ef1739cbe3dc17a8418792d3fb999f"
server_key_iv: "4b8a0ed7b6826be32ad5ed6f8d3f4bfd"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:30.743716 11931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "df99faf97aa74d04b7988b7e475ae24c"
format_stamp: "Formatted at 2026-09-26 23:25:30 on dist-test-slave-q5g8"
server_key: "e3ef1739cbe3dc17a8418792d3fb999f"
server_key_iv: "4b8a0ed7b6826be32ad5ed6f8d3f4bfd"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:30.759042 11931 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.012s sys 0.000s
I20260926 23:25:30.794437 11950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:30.797135 11931 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.015s sys 0.001s
I20260926 23:25:30.797399 11931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "df99faf97aa74d04b7988b7e475ae24c"
format_stamp: "Formatted at 2026-09-26 23:25:30 on dist-test-slave-q5g8"
server_key: "e3ef1739cbe3dc17a8418792d3fb999f"
server_key_iv: "4b8a0ed7b6826be32ad5ed6f8d3f4bfd"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:30.797735 11931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:25:30.832655 11931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:25:30.838370 11931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:25:30.838809 11931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:25:30.840121 11931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:25:30.842067 11931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:25:30.842204 11931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:25:30.842355 11931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:25:30.842440 11931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:30.889470 11931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38541
I20260926 23:25:30.889472 12067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38541 every 8 connection(s)
I20260926 23:25:30.891021 11931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:25:30.893918 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 11931
I20260926 23:25:30.894112 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:25:30.894654 388 external_mini_cluster.cc:1428] Setting key c9c53d13e1c9f63d826badb8f9d1b3b5
Sep 26 23:25:30 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:25:30.925746 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:30.895472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47223 (local address 127.0.97.62:43591)
0926 23:25:30.905225 (+ 9753us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:30.905232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:30.906326 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 23:25:30.909213 (+ 2887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:30.909220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:30.909341 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:30.909479 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:30.911288 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:30.912095 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:30.914011 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:30.914544 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:30.914688 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:30.919171 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:30.919178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:30.919182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:30.919232 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:30.922021 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:30.922974 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:30.922979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:30.922981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:30.923184 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:30.923728 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:30.923733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:30.923737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:30.924044 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:30.924208 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:30.925118 (+ 910us) server_negotiation.cc:299] Negotiation successful
0926 23:25:30.925360 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9565,"thread_start_us":9375,"threads_started":1}
I20260926 23:25:30.927002 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:30.895560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43591 (local address 127.0.97.2:47223)
0926 23:25:30.905732 (+ 10172us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:30.905803 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:30.908804 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:30.909820 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:30.909831 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:30.910341 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:25:30.911080 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:30.911104 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:30.912300 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:30.912304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:30.913829 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:30.913852 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:30.914812 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:30.914820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:30.915093 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:30.915972 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:25:30.915997 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:25:30.918880 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:25:30.922221 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:25:30.922231 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:25:30.922267 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:25:30.922816 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:25:30.923342 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:25:30.923347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:25:30.923351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:25:30.923595 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:25:30.924185 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:25:30.924194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:25:30.924633 (+ 439us) client_negotiation.cc:769] Sending connection context
0926 23:25:30.925009 (+ 376us) client_negotiation.cc:240] Negotiation successful
0926 23:25:30.925604 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9789,"thread_start_us":9509,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:25:30.928813 12068 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43591
I20260926 23:25:30.929353 12068 heartbeater.cc:464] Registering TS with master...
I20260926 23:25:30.930379 12068 heartbeater.cc:511] Master 127.0.97.62:43591 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:25:30.932482 11712 ts_manager.cc:194] Registered new tserver with Master: df99faf97aa74d04b7988b7e475ae24c (127.0.97.2:38541)
I20260926 23:25:30.933818 11712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:25:30.969298 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43591
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:25:31.035880 11925 heartbeater.cc:503] Master 127.0.97.62:43591 was elected leader, sending a full tablet report...
W20260926 23:25:31.360499 12074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:25:31.360878 12074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:25:31.361063 12074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:25:31.368475 12074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:25:31.368602 12074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:25:31.368873 12074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:25:31.379091 12074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:42989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:25:31.381064 12074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:25:31.382511 12074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:25:31.397565 12082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:31.397665 12080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:25:31.397591 12079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:25:31.398010 12074 server_base.cc:1034] running on GCE node
I20260926 23:25:31.398819 12074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:25:31.399935 12074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:25:31.401157 12074 hybrid_clock.cc:648] HybridClock initialized: now 1790465131401091 us; error 36 us; skew 500 ppm
I20260926 23:25:31.401475 12074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:25:31 dist-test-slave-q5g8 krb5kdc[11236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:25:31.404875 12074 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:25:31.406930 12074 webserver.cc:460] Webserver started at http://127.0.97.3:37165/ using document root <none> and password file <none>
I20260926 23:25:31.407685 12074 fs_manager.cc:359] Metadata directory not provided
I20260926 23:25:31.407836 12074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:25:31.408180 12074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:25:31 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:25:31.425148 12074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "7d6efa62d5284a889a125e8148a8616d"
format_stamp: "Formatted at 2026-09-26 23:25:31 on dist-test-slave-q5g8"
server_key: "e6505dc2195e0e0e632c2272f9346143"
server_key_iv: "5c152d5c9b05378aecdf471c08c32ffc"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:31.426502 12074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7d6efa62d5284a889a125e8148a8616d"
format_stamp: "Formatted at 2026-09-26 23:25:31 on dist-test-slave-q5g8"
server_key: "e6505dc2195e0e0e632c2272f9346143"
server_key_iv: "5c152d5c9b05378aecdf471c08c32ffc"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:31.435639 12074 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260926 23:25:31.453526 12089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:31.455224 12074 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:25:31.455430 12074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "7d6efa62d5284a889a125e8148a8616d"
format_stamp: "Formatted at 2026-09-26 23:25:31 on dist-test-slave-q5g8"
server_key: "e6505dc2195e0e0e632c2272f9346143"
server_key_iv: "5c152d5c9b05378aecdf471c08c32ffc"
server_key_version: "kuduclusterkey@0"
I20260926 23:25:31.455670 12074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:25:31.486824 12074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:25:31.491093 12074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:25:31.491451 12074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:25:31.493016 12074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:25:31.494556 12074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:25:31.494673 12074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:31.494797 12074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:25:31.494879 12074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:25:31.533880 12074 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42635
I20260926 23:25:31.533933 12201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42635 every 8 connection(s)
I20260926 23:25:31.535312 12074 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:25:31.542080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 12074
I20260926 23:25:31.542289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:25:31.542754 388 external_mini_cluster.cc:1428] Setting key cc7a77e83374242449060858d31e4b69
Sep 26 23:25:31 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:25:31.557364 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.538100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50159 (local address 127.0.97.62:43591)
0926 23:25:31.538590 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.538595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.539550 (+ 955us) server_negotiation.cc:407] Connection header received
0926 23:25:31.541553 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.541559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.541699 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.541833 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:31.543640 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.544791 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.547156 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.547592 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.547702 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.551162 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:31.551168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:31.551171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:31.551208 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:31.554067 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:31.554882 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:31.554887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:31.554890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:31.555038 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:31.555465 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:31.555469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:31.555471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:31.555811 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:31.555937 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.556662 (+ 725us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.556925 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":265,"threads_started":1}
I20260926 23:25:31.558322 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.538598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43591 (local address 127.0.97.3:50159)
0926 23:25:31.539381 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:31.539422 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:31.541249 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:31.542073 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:31.542083 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:31.542633 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:25:31.543319 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.543339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.545002 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.545014 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.547022 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.547036 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.547808 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.547813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.548019 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.548727 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:25:31.548770 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:25:31.550901 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:25:31.554230 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:25:31.554238 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:25:31.554252 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:25:31.554740 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:25:31.555156 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:25:31.555159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:25:31.555162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:25:31.555360 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:25:31.555953 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:25:31.555960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:25:31.556280 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 23:25:31.556523 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 23:25:31.556988 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":323,"threads_started":1}
I20260926 23:25:31.559890 12202 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43591
I20260926 23:25:31.560294 12202 heartbeater.cc:464] Registering TS with master...
I20260926 23:25:31.561255 12202 heartbeater.cc:511] Master 127.0.97.62:43591 requested a full tablet report, sending...
I20260926 23:25:31.562862 11711 ts_manager.cc:194] Registered new tserver with Master: 7d6efa62d5284a889a125e8148a8616d (127.0.97.3:42635)
I20260926 23:25:31.563822 11711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50159
I20260926 23:25:31.572819 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:25:31.594031 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.579015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.97.62:43591)
0926 23:25:31.579240 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.579246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.579844 (+ 598us) server_negotiation.cc:407] Connection header received
0926 23:25:31.580017 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.580022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.580128 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.580237 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:31.582193 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.582987 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.584595 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.585241 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.585392 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.588054 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:31.588060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:31.588064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:31.588106 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:31.591085 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:31.591975 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:31.591979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:31.591982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:31.592122 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:31.592572 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:31.592575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:31.592577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:31.592946 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:31.593141 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.593554 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.593794 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 23:25:31.599368 11711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:25:31.608079 11711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:25:31.636548 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.622419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42635 (local address 127.0.0.1:48882)
0926 23:25:31.623742 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:31.623792 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:31.624000 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:31.626574 (+ 2574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:31.626583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:31.626671 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:25:31.627883 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.627899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.629876 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.629881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.633022 (+ 3141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.633036 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.635567 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.635574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.635849 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.635891 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:25:31.636034 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:25:31.636182 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":9,"spinlock_wait_cycles":32640,"thread_start_us":542,"threads_started":1}
I20260926 23:25:31.636744 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.623539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.0.97.3:42635)
0926 23:25:31.625804 (+ 2265us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.625814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.625852 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:25:31.625962 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.625970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.626282 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.626524 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:25:31.628096 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.629656 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.633195 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.635264 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.635548 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.635793 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.636111 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.636296 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2048,"thread_start_us":1311,"threads_started":1}
I20260926 23:25:31.637606 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.622800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38541 (local address 127.0.0.1:48176)
0926 23:25:31.624830 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:31.624855 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:31.625484 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:31.626564 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:31.626575 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:31.626647 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:25:31.627778 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.627801 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.629774 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.629781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.632872 (+ 3091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.632883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.636654 (+ 3771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.636661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.636875 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.636905 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:25:31.636995 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 23:25:31.637136 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"mutex_wait_us":600,"spinlock_wait_cycles":4352,"thread_start_us":4520,"threads_started":1}
I20260926 23:25:31.637657 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.624167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.0.97.2:38541)
0926 23:25:31.626007 (+ 1840us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.626015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.626040 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:25:31.626132 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.626140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.626324 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.627103 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:25:31.628042 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.629624 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.633057 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.636391 (+ 3334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.636539 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.636764 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.637078 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.637268 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1694,"thread_start_us":1568,"threads_started":1}
I20260926 23:25:31.638094 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.623283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44205 (local address 127.0.0.1:56776)
0926 23:25:31.627835 (+ 4552us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:31.627864 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:31.627993 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:31.628812 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:31.628817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:31.628844 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:25:31.629249 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.629261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.631583 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.631588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.633601 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.633612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.637177 (+ 3565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.637184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.637380 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.637412 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:25:31.637518 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:25:31.637699 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4371,"mutex_wait_us":557,"thread_start_us":633,"threads_started":1}
I20260926 23:25:31.638742 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.623621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.97.1:44205)
0926 23:25:31.624369 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.624383 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.628143 (+ 3760us) server_negotiation.cc:407] Connection header received
0926 23:25:31.628268 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.628276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.628556 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.628808 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:25:31.629455 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.631305 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.633764 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.635250 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.635444 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.635730 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.638174 (+ 2444us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.638390 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":385,"threads_started":1}
I20260926 23:25:31.641964 12003 tablet_service.cc:1467] Processing CreateTablet for tablet 074db0a05b4647a1a23872138d69b869 (DEFAULT_TABLE table=test-table [id=09b0d43fd0d8439baf994b563c7206c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:25:31.642359 11860 tablet_service.cc:1467] Processing CreateTablet for tablet 074db0a05b4647a1a23872138d69b869 (DEFAULT_TABLE table=test-table [id=09b0d43fd0d8439baf994b563c7206c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:25:31.642189 12137 tablet_service.cc:1467] Processing CreateTablet for tablet 074db0a05b4647a1a23872138d69b869 (DEFAULT_TABLE table=test-table [id=09b0d43fd0d8439baf994b563c7206c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:25:31.643378 12003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074db0a05b4647a1a23872138d69b869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:25:31.643726 12137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074db0a05b4647a1a23872138d69b869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:25:31.643726 11860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074db0a05b4647a1a23872138d69b869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:25:31.656127 12221 tablet_bootstrap.cc:492] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Bootstrap starting.
I20260926 23:25:31.658799 12222 tablet_bootstrap.cc:492] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Bootstrap starting.
I20260926 23:25:31.662228 12221 tablet_bootstrap.cc:654] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Neither blocks nor log segments found. Creating new log.
I20260926 23:25:31.662539 12222 tablet_bootstrap.cc:654] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Neither blocks nor log segments found. Creating new log.
I20260926 23:25:31.664312 12221 log.cc:824] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:25:31.664312 12222 log.cc:824] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:25:31.665982 12223 tablet_bootstrap.cc:492] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Bootstrap starting.
I20260926 23:25:31.671231 12222 tablet_bootstrap.cc:492] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: No bootstrap required, opened a new log
I20260926 23:25:31.671609 12222 ts_tablet_manager.cc:1397] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260926 23:25:31.671687 12223 tablet_bootstrap.cc:654] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Neither blocks nor log segments found. Creating new log.
I20260926 23:25:31.673152 12223 log.cc:824] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Log is configured to *not* fsync() on all Append() calls
I20260926 23:25:31.674367 12221 tablet_bootstrap.cc:492] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: No bootstrap required, opened a new log
I20260926 23:25:31.674772 12221 ts_tablet_manager.cc:1397] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20260926 23:25:31.676409 12223 tablet_bootstrap.cc:492] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: No bootstrap required, opened a new log
I20260926 23:25:31.676765 12223 ts_tablet_manager.cc:1397] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260926 23:25:31.683914 12222 raft_consensus.cc:348] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.684489 12222 raft_consensus.cc:374] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:25:31.684643 12222 raft_consensus.cc:729] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d6efa62d5284a889a125e8148a8616d, State: Initialized, Role: FOLLOWER
I20260926 23:25:31.685290 12223 raft_consensus.cc:348] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.685451 12222 consensus_queue.cc:260] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.685684 12223 raft_consensus.cc:374] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:25:31.685814 12223 raft_consensus.cc:729] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c96efd3a0d4556a6e4f432f4e99076, State: Initialized, Role: FOLLOWER
I20260926 23:25:31.686296 12221 raft_consensus.cc:348] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.686774 12202 heartbeater.cc:503] Master 127.0.97.62:43591 was elected leader, sending a full tablet report...
I20260926 23:25:31.686599 12223 consensus_queue.cc:260] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.686786 12221 raft_consensus.cc:374] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:25:31.686908 12221 raft_consensus.cc:729] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df99faf97aa74d04b7988b7e475ae24c, State: Initialized, Role: FOLLOWER
I20260926 23:25:31.687582 12222 ts_tablet_manager.cc:1428] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260926 23:25:31.687614 12221 consensus_queue.cc:260] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.688794 12068 heartbeater.cc:503] Master 127.0.97.62:43591 was elected leader, sending a full tablet report...
I20260926 23:25:31.689459 12221 ts_tablet_manager.cc:1428] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 23:25:31.692886 12223 ts_tablet_manager.cc:1428] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260926 23:25:31.739965 11929 tablet.cc:2369] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:25:31.787098 12203 tablet.cc:2369] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:25:31.804044 12227 raft_consensus.cc:482] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:25:31.804358 12227 raft_consensus.cc:504] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.806732 12227 leader_election.cc:288] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df99faf97aa74d04b7988b7e475ae24c (127.0.97.2:38541), f0c96efd3a0d4556a6e4f432f4e99076 (127.0.97.1:44205)
I20260926 23:25:31.813462 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.807198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33343 (local address 127.0.97.2:38541)
0926 23:25:31.807428 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.807435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.807596 (+ 161us) server_negotiation.cc:407] Connection header received
0926 23:25:31.807895 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.807900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.808003 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.808114 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:25:31.808580 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.809543 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.811438 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.812563 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.812668 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.812805 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.813165 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.813261 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 23:25:31.813619 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.807085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38541 (local address 127.0.97.3:33343)
0926 23:25:31.807451 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:31.807475 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:31.807674 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:31.808167 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:31.808172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:31.808203 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:25:31.808445 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.808457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.809673 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.809678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.811276 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.811287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.812774 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.812781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.812978 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.813015 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:25:31.813127 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 23:25:31.813284 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":1}
I20260926 23:25:31.814421 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.807181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44205 (local address 127.0.97.3:44623)
0926 23:25:31.808052 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0926 23:25:31.808072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:25:31.808192 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:25:31.808580 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:25:31.808584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:25:31.808608 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:25:31.808986 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.808999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.810398 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.810402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.812052 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:25:31.812068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.813744 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.813750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:25:31.813917 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.813950 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:25:31.814046 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:25:31.814184 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":504,"threads_started":1}
I20260926 23:25:31.814529 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.807271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44623 (local address 127.0.97.1:44205)
0926 23:25:31.807527 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.807531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.808169 (+ 638us) server_negotiation.cc:407] Connection header received
0926 23:25:31.808348 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.808353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.808432 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.808553 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:25:31.809205 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.810242 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.812291 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.813450 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.813584 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.813701 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.814109 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.814261 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 23:25:31.815096 12023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074db0a05b4647a1a23872138d69b869" candidate_uuid: "7d6efa62d5284a889a125e8148a8616d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df99faf97aa74d04b7988b7e475ae24c" is_pre_election: true
I20260926 23:25:31.815717 12023 raft_consensus.cc:2454] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d6efa62d5284a889a125e8148a8616d in term 0.
I20260926 23:25:31.815635 11880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074db0a05b4647a1a23872138d69b869" candidate_uuid: "7d6efa62d5284a889a125e8148a8616d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" is_pre_election: true
I20260926 23:25:31.816290 11880 raft_consensus.cc:2454] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d6efa62d5284a889a125e8148a8616d in term 0.
I20260926 23:25:31.816576 12091 leader_election.cc:302] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d6efa62d5284a889a125e8148a8616d, df99faf97aa74d04b7988b7e475ae24c; no voters:
I20260926 23:25:31.817117 12227 raft_consensus.cc:2781] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:25:31.817267 12227 raft_consensus.cc:482] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:25:31.817369 12227 raft_consensus.cc:3037] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:25:31.820127 12227 raft_consensus.cc:504] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.820925 12227 leader_election.cc:288] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [CANDIDATE]: Term 1 election: Requested vote from peers df99faf97aa74d04b7988b7e475ae24c (127.0.97.2:38541), f0c96efd3a0d4556a6e4f432f4e99076 (127.0.97.1:44205)
I20260926 23:25:31.821499 12023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074db0a05b4647a1a23872138d69b869" candidate_uuid: "7d6efa62d5284a889a125e8148a8616d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df99faf97aa74d04b7988b7e475ae24c"
I20260926 23:25:31.821507 11880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074db0a05b4647a1a23872138d69b869" candidate_uuid: "7d6efa62d5284a889a125e8148a8616d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c96efd3a0d4556a6e4f432f4e99076"
I20260926 23:25:31.821728 12023 raft_consensus.cc:3037] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:25:31.821741 11880 raft_consensus.cc:3037] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:25:31.824260 12023 raft_consensus.cc:2454] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d6efa62d5284a889a125e8148a8616d in term 1.
I20260926 23:25:31.824270 11880 raft_consensus.cc:2454] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d6efa62d5284a889a125e8148a8616d in term 1.
I20260926 23:25:31.824802 12091 leader_election.cc:302] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d6efa62d5284a889a125e8148a8616d, df99faf97aa74d04b7988b7e475ae24c; no voters:
I20260926 23:25:31.825183 12227 raft_consensus.cc:2781] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:25:31.825677 12227 raft_consensus.cc:686] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 1 LEADER]: Becoming Leader. State: Replica: 7d6efa62d5284a889a125e8148a8616d, State: Running, Role: LEADER
I20260926 23:25:31.826232 12227 consensus_queue.cc:237] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } }
I20260926 23:25:31.832124 11711 catalog_manager.cc:5495] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d6efa62d5284a889a125e8148a8616d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7d6efa62d5284a889a125e8148a8616d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d6efa62d5284a889a125e8148a8616d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 } health_report { overall_health: UNKNOWN } } }
I20260926 23:25:31.882151 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.876089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.0.97.3:42635)
0926 23:25:31.876270 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.876275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.876363 (+ 88us) server_negotiation.cc:407] Connection header received
0926 23:25:31.876623 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.876626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.876743 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.876880 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:25:31.877223 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.877937 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.879955 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.880347 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.880460 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.881017 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:25:31.881591 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:25:31.881711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.881867 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.881940 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:25:31.891593 12023 raft_consensus.cc:1261] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 1 FOLLOWER]: Refusing update from remote peer 7d6efa62d5284a889a125e8148a8616d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:25:31.891590 11880 raft_consensus.cc:1261] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 1 FOLLOWER]: Refusing update from remote peer 7d6efa62d5284a889a125e8148a8616d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:25:31.892776 12227 consensus_queue.cc:1035] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df99faf97aa74d04b7988b7e475ae24c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:25:31.893218 12231 consensus_queue.cc:1035] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0c96efd3a0d4556a6e4f432f4e99076" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:25:31.894117 12069 tablet.cc:2369] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:25:31.915810 12235 mvcc.cc:205] Tried to move back new op lower bound from 7333745180213723136 to 7333745179963789312. Current Snapshot: MvccSnapshot[applied={T|T < 7333745180213723136}]
I20260926 23:25:31.934499 11710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:25:31.934845 11710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:25:31.939884 11710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d6b7f15dac7492281aa76660ce288ff: Sending DeleteTablet for 3 replicas of tablet 074db0a05b4647a1a23872138d69b869
I20260926 23:25:31.941613 12003 tablet_service.cc:1514] Processing DeleteTablet for tablet 074db0a05b4647a1a23872138d69b869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:25:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48176
I20260926 23:25:31.941576 12137 tablet_service.cc:1514] Processing DeleteTablet for tablet 074db0a05b4647a1a23872138d69b869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:25:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48882
I20260926 23:25:31.942776 11860 tablet_service.cc:1514] Processing DeleteTablet for tablet 074db0a05b4647a1a23872138d69b869 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:25:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56776
I20260926 23:25:31.943755 12246 tablet_replica.cc:330] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: stopping tablet replica
I20260926 23:25:31.944111 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 11792
I20260926 23:25:31.944366 12246 raft_consensus.cc:2229] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:25:31.945019 12246 raft_consensus.cc:2258] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:25:31.947705 12246 ts_tablet_manager.cc:1905] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:25:31.952522 12244 tablet_replica.cc:330] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: stopping tablet replica
I20260926 23:25:31.953239 12244 raft_consensus.cc:2229] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:25:31.953969 12244 raft_consensus.cc:2258] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:25:31.957816 12246 ts_tablet_manager.cc:1918] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:25:31.958124 12246 log.cc:1196] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/074db0a05b4647a1a23872138d69b869
I20260926 23:25:31.958844 12246 ts_tablet_manager.cc:1939] T 074db0a05b4647a1a23872138d69b869 P f0c96efd3a0d4556a6e4f432f4e99076: Deleting consensus metadata
I20260926 23:25:31.958873 12244 ts_tablet_manager.cc:1905] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:25:31.961238 11699 catalog_manager.cc:4841] TS f0c96efd3a0d4556a6e4f432f4e99076 (127.0.97.1:44205): tablet 074db0a05b4647a1a23872138d69b869 (table test-table [id=09b0d43fd0d8439baf994b563c7206c4]) successfully deleted
I20260926 23:25:31.965451 12245 tablet_replica.cc:330] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: stopping tablet replica
I20260926 23:25:31.966079 12245 raft_consensus.cc:2229] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:25:31.966981 12245 raft_consensus.cc:2258] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 23:25:31 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:25:31.969297 12244 ts_tablet_manager.cc:1918] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:25:31.972100 12245 ts_tablet_manager.cc:1905] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:25:31.972858 12244 log.cc:1196] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/074db0a05b4647a1a23872138d69b869
I20260926 23:25:31.973549 12244 ts_tablet_manager.cc:1939] T 074db0a05b4647a1a23872138d69b869 P df99faf97aa74d04b7988b7e475ae24c: Deleting consensus metadata
I20260926 23:25:31.977182 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:31.946123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.0.97.1:44205)
0926 23:25:31.946318 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:31.946330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:31.954940 (+ 8610us) server_negotiation.cc:407] Connection header received
0926 23:25:31.955151 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:31.955156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:31.955267 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:31.955402 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:31.957199 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.958393 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.961588 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:31.962190 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:31.962297 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:31.968900 (+ 6603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:31.968919 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:31.968940 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:31.968998 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:31.973388 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:31.974410 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:31.974421 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:31.974424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:31.974615 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:31.975057 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:31.975063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:31.975066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:31.975567 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:31.975858 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:31.976438 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 23:25:31.976744 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:25:31.978729 11920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:25:31.982278 12245 ts_tablet_manager.cc:1918] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:25:31.982515 12245 log.cc:1196] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/074db0a05b4647a1a23872138d69b869
I20260926 23:25:31.982999 12245 ts_tablet_manager.cc:1939] T 074db0a05b4647a1a23872138d69b869 P 7d6efa62d5284a889a125e8148a8616d: Deleting consensus metadata
I20260926 23:25:31.984831 11698 catalog_manager.cc:4841] TS 7d6efa62d5284a889a125e8148a8616d (127.0.97.3:42635): tablet 074db0a05b4647a1a23872138d69b869 (table test-table [id=09b0d43fd0d8439baf994b563c7206c4]) successfully deleted
I20260926 23:25:31.986217 11697 catalog_manager.cc:4841] TS df99faf97aa74d04b7988b7e475ae24c (127.0.97.2:38541): tablet 074db0a05b4647a1a23872138d69b869 (table test-table [id=09b0d43fd0d8439baf994b563c7206c4]) successfully deleted
I20260926 23:25:32.082273 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 11792
I20260926 23:25:32.100776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 11931
Sep 26 23:25:32 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:25:32.119937 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:32.101861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.0.97.2:38541)
0926 23:25:32.102110 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:32.102118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:32.102143 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:25:32.102388 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:32.102404 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:32.102484 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:32.102658 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:32.104231 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:32.105378 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:32.106965 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:32.107417 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:32.107579 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:32.112450 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:32.112469 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:32.112486 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:32.112528 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:32.116226 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:32.117345 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:32.117354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:32.117357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:32.117524 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:32.118002 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:32.118009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:32.118015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:32.118460 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:32.118820 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:32.119239 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 23:25:32.119500 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:25:32.120816 12063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:25:32.232394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 11931
I20260926 23:25:32.251492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 12074
Sep 26 23:25:32 dist-test-slave-q5g8 krb5kdc[11236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:25:32.271617 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:25:32.252534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.97.3:42635)
0926 23:25:32.252799 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:25:32.252808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:25:32.252835 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:25:32.252934 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:25:32.252940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:25:32.253029 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:25:32.253151 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:25:32.254570 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:32.256050 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:32.257789 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:25:32.258636 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:25:32.258824 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:25:32.264551 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:25:32.264574 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:25:32.264592 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:25:32.264644 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:25:32.268140 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:32.269201 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:32.269212 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:32.269214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:32.269380 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:25:32.269860 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:25:32.269864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:25:32.269866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:25:32.270287 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:25:32.270627 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 23:25:32.270988 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 23:25:32.271248 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:25:32.272542 12197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:25:32.408146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 12074
I20260926 23:25:32.429034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 11680
I20260926 23:25:32.430419 11742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:25:32.549005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 11680
I20260926 23:25:32.573499 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 23:25:32.909608 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 23:25:32.909847 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 23:25:33.256453 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 23:25:33.256 UTC [11266] LOG: received smart shutdown request
2026-09-26 23:25:33.260 UTC [11266] LOG: background worker "logical replication launcher" (PID 11275) exited with exit code 1
2026-09-26 23:25:33.261 UTC [11270] LOG: shutting down
2026-09-26 23:25:33.471 UTC [11266] LOG: database system is shut down
2026-09-26T23:25:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53548 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:25:33 dist-test-slave-q5g8 krb5kdc[12260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:25:33 dist-test-slave-q5g8 krb5kdc[12260](info): set up 2 sockets
Sep 26 23:25:33 dist-test-slave-q5g8 krb5kdc[12260](info): commencing operation
krb5kdc: starting...
W20260926 23:25:34.757098 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.011s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:25:34 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:25:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 23:25:35.115811 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknM1jIl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/postgres -l logfile start
2026-09-26 23:25:37.921 UTC [12290] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 23:25:37.921 UTC [12290] LOG: listening on IPv4 address "127.0.97.20", port 50281
2026-09-26 23:25:37.925 UTC [12290] LOG: listening on Unix socket "/tmp/.s.PGSQL.50281"
2026-09-26 23:25:37.949 UTC [12293] LOG: database system was shut down at 2026-09-26 23:25:36 UTC
2026-09-26 23:25:37.954 UTC [12290] LOG: database system is ready to accept connections
I20260926 23:25:38.964651 388 mini_postgres.cc:96] Postgres bound to 50281
2026-09-26 23:25:38.982 UTC [12301] FATAL: database "slave" does not exist
127.0.97.20:50281 - accepting connections
I20260926 23:25:38.983036 388 mini_ranger.cc:161] Starting Ranger...
I20260926 23:25:39.009902 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 23:25:39.549333 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 23:25:39.549467 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-admin
2026-09-26 23:25:39,850 [I] DB FLAVOR :POSTGRES
2026-09-26 23:25:39,851 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:25:39,851 [I] Checking connection..
2026-09-26 23:25:39,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 23:25:40,106 [I] Checking connection passed.
2026-09-26 23:25:40,107 [I] --------- Verifying version history table ---------
2026-09-26 23:25:40,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:25:40,370 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:25:40,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:25:40,614 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:25:40,615 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 23:25:40,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 23:25:40.859 UTC [12365] WARNING: there is no transaction in progress
2026-09-26 23:25:40,866 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 23:25:40,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:25:41,123 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 23:25:41,124 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:25:41,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 23:25:41,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 23:25:41,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q5g8.c.gcp-upstream.internal','N') ;"
2026-09-26 23:25:41,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:25:42,125 [I] Table x_portal_user does not exist in database ranger
2026-09-26 23:25:42,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:25:42,381 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 23:25:42,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:25:42,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:25:42,900 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 23:25:42,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 23:25:44.023 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.043 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.065 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.084 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.104 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.129 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.202 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.211 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.228 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.239 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.250 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.269 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.280 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.289 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.298 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:44.307 UTC [12473] WARNING: there is no transaction in progress
2026-09-26 23:25:45,067 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 23:25:45,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 23:25:45,334 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 23:25:45,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:25:45,601 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 23:25:45,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:25:45,864 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 23:25:45,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:25:46,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:25:46,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q5g8.c.gcp-upstream.internal';"
2026-09-26 23:25:46,655 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 23:25:46,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 23:25:46,928 [I] DB_PATCHES have already been applied
I20260926 23:25:46.935057 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknM1jIl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 23:25:46.935158 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 23:25:46.951297 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:32935
Sep 26, 2026 11:25:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:25:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:25:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:25:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:25:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:25:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:25:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 23:25:48.154659 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.172s user 0.000s sys 0.040s
Sep 26, 2026 11:26:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:26:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 11:26:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:26:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:26:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 23:26:11.733668 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 23:26:11 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465171, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 23:26:11.899068 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 23:26:11.919775 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 23:26:12.894368 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 23:26:12.894488 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms
I20260926 23:26:12.894548 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:50281
2026-09-26 23:26:13,248 [I] DB FLAVOR :POSTGRES
2026-09-26 23:26:13,249 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:26:13,249 [I] Checking connection
2026-09-26 23:26:13,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 23:26:13,481 [I] connection success
2026-09-26 23:26:13,481 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 23:26:13,481 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 23:26:13,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 23:26:13,751 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 23:26:13,751 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:26:13,751 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 23:26:13,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 23:26:14,045 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 23:26:14.548975 388 mini_ranger_kms.cc:310] Created kms service
I20260926 23:26:14.724036 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 23:26:14.770215 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 23:26:14 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465171, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:15.199136 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 23:26:15.199445 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknM1jIl/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknM1jIl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 23:26:15.199505 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 23:26:15.219839 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12654
I20260926 23:26:15.219972 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:41887
Sep 26, 2026 11:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 23:26:17 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465177, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:17.555958 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.657s user 0.070s sys 0.300s
I20260926 23:26:17.556213 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 23:26:17.556383 388 mini_ranger_kms.cc:401] 127.0.97.20:41887/kms/v1/keys
Sep 26 23:26:17 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465177, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:26:18 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465177, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:26:20 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:26:20.741910 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43199
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43199
--ranger_config_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:21.112313 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:21.112632 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:21.112803 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:21.120395 12704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:21.120525 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:21.120573 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:21.120632 12704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:21.120748 12704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:21.131475 12704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:41887/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43199
--ranger_config_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:21.133345 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:21.134907 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:21.147928 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:21.148063 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:21.148020 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:21.148583 12704 server_base.cc:1034] running on GCE node
I20260926 23:26:21.149530 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:21.151077 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:21.152319 12704 hybrid_clock.cc:648] HybridClock initialized: now 1790465181152281 us; error 36 us; skew 500 ppm
I20260926 23:26:21.152740 12704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:21 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:21.156500 12704 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:26:21.158133 12704 webserver.cc:460] Webserver started at http://127.0.97.62:41075/ using document root <none> and password file <none>
I20260926 23:26:21.158859 12704 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:21.159003 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:21.159335 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:26:21 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:21.226212 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "36845a2d41174672891df711ac84ad19"
format_stamp: "Formatted at 2026-09-26 23:26:21 on dist-test-slave-q5g8"
server_key: "e7b818f21b5c1977b5cdc2ae8c80bf03"
server_key_iv: "522e13e8eaaf73fb1c9d3bcb254ec328"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:21.227173 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "36845a2d41174672891df711ac84ad19"
format_stamp: "Formatted at 2026-09-26 23:26:21 on dist-test-slave-q5g8"
server_key: "e7b818f21b5c1977b5cdc2ae8c80bf03"
server_key_iv: "522e13e8eaaf73fb1c9d3bcb254ec328"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:21.233215 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:26:21.254060 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:21.255743 12704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:26:21.255973 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "36845a2d41174672891df711ac84ad19"
format_stamp: "Formatted at 2026-09-26 23:26:21 on dist-test-slave-q5g8"
server_key: "e7b818f21b5c1977b5cdc2ae8c80bf03"
server_key_iv: "522e13e8eaaf73fb1c9d3bcb254ec328"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:21.256248 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:21.271840 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:21.276402 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:21.276872 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:21.297010 12704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43199
I20260926 23:26:21.296983 12770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43199 every 8 connection(s)
I20260926 23:26:21.298578 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:26:21.299393 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 12704
I20260926 23:26:21.299620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:26:21.300395 388 external_mini_cluster.cc:1428] Setting key cd9232d83176335d9fe7e884a6aa9529
I20260926 23:26:21.303246 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:26:21 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:21.320448 12771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Bootstrap starting.
I20260926 23:26:21.324271 12771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:21.325543 12771 log.cc:824] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:21.329228 12771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: No bootstrap required, opened a new log
I20260926 23:26:21.329782 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:21.302904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.0.97.62:43199)
0926 23:26:21.305314 (+ 2410us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:21.305333 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:21.305860 (+ 527us) server_negotiation.cc:407] Connection header received
0926 23:26:21.309113 (+ 3253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:21.309151 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:21.309876 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:21.310368 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:21.312216 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:21.313744 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:21.315718 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:21.316542 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:21.316788 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:21.321626 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:21.321644 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:21.321661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:21.321710 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:21.325125 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:21.325949 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:21.325960 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:21.325972 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:21.326255 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:21.326725 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:21.326742 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:21.326751 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:21.327416 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:21.327646 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:21.328169 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 23:26:21.328557 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2119,"thread_start_us":1849,"threads_started":1}
I20260926 23:26:21.337989 12771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:21.338421 12771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:21.338536 12771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36845a2d41174672891df711ac84ad19, State: Initialized, Role: FOLLOWER
I20260926 23:26:21.339147 12771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:21.339427 12771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:26:21.339578 12771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:26:21.339771 12771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:21.342545 12771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:21.343093 12771 leader_election.cc:302] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36845a2d41174672891df711ac84ad19; no voters:
I20260926 23:26:21.343689 12771 leader_election.cc:288] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:26:21.343832 12776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:26:21.344547 12776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 LEADER]: Becoming Leader. State: Replica: 36845a2d41174672891df711ac84ad19, State: Running, Role: LEADER
I20260926 23:26:21.345193 12776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:21.345928 12771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:21.348132 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36845a2d41174672891df711ac84ad19. Latest consensus state: current_term: 1 leader_uuid: "36845a2d41174672891df711ac84ad19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } } }
I20260926 23:26:21.348191 12777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36845a2d41174672891df711ac84ad19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } } }
I20260926 23:26:21.348573 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:21.348574 12777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:21.351624 12782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:26:21.356082 12782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:26:21.359778 12771 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 23:26:21.374995 12782 catalog_manager.cc:1347] Generated new cluster ID: dcb16e442617431dbf568ad73a22814b
I20260926 23:26:21.375171 12782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:26:21.389083 12782 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:26:21.390542 12782 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:26:21.414806 12782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Generated new TSK 0
I20260926 23:26:21.416138 12782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:26:22.214700 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:26:22 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 23:26:22.691720 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:22.692157 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:22.692380 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:22.705142 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:22.705384 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:22.705587 12816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:26:22.721745 12816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:41887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:22.723697 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:22.725543 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:22.747641 12826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:22.748747 12827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:22.756954 12829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:22.757710 12816 server_base.cc:1034] running on GCE node
I20260926 23:26:22.758594 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:22.760016 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:22.761349 12816 hybrid_clock.cc:648] HybridClock initialized: now 1790465182761251 us; error 71 us; skew 500 ppm
I20260926 23:26:22.761825 12816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:22 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:22.766242 12816 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:26:22.770118 12816 webserver.cc:460] Webserver started at http://127.0.97.1:44959/ using document root <none> and password file <none>
I20260926 23:26:22.771003 12816 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:22.771142 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:22.771492 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:26:22 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:22.793591 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "b99fd6eead1740efa66d7dce953884fe"
format_stamp: "Formatted at 2026-09-26 23:26:22 on dist-test-slave-q5g8"
server_key: "a5b2edc340bbfd34b3e596463690d014"
server_key_iv: "4172ee58307eac50afe02746ad3ac10a"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:22.794920 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b99fd6eead1740efa66d7dce953884fe"
format_stamp: "Formatted at 2026-09-26 23:26:22 on dist-test-slave-q5g8"
server_key: "a5b2edc340bbfd34b3e596463690d014"
server_key_iv: "4172ee58307eac50afe02746ad3ac10a"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:22.822695 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.027s user 0.000s sys 0.016s
I20260926 23:26:22.850252 12836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:22.852584 12816 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.007s
I20260926 23:26:22.852927 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "b99fd6eead1740efa66d7dce953884fe"
format_stamp: "Formatted at 2026-09-26 23:26:22 on dist-test-slave-q5g8"
server_key: "a5b2edc340bbfd34b3e596463690d014"
server_key_iv: "4172ee58307eac50afe02746ad3ac10a"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:22.853317 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:22.897965 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:22.903756 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:22.904193 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:22.906821 12816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:22.909292 12816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:22.909452 12816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:26:22.909655 12816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:22.909777 12816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:26:22 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:23.115919 12816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41233
I20260926 23:26:23.115981 12948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41233 every 8 connection(s)
I20260926 23:26:23.117733 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:26:23.119522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 12816
I20260926 23:26:23.119781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:26:23.120656 388 external_mini_cluster.cc:1428] Setting key 8f98c7e96a91d71e99cfbc6c1cbafa3e
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 26 23:26:23 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:23.177522 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:23.122609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.1:41011)
0926 23:26:23.150087 (+ 27478us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:23.150162 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:23.153269 (+ 3107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:23.154496 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:23.154514 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:23.155309 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:23.156583 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:23.156623 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.158175 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.158181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:23.160772 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:23.160796 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.162587 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.162596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:23.162966 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:23.163980 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:23.164016 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:23.167086 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:23.171469 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:23.171482 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:23.171504 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:23.172149 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:23.172882 (+ 733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:23.172889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:23.172893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:23.173250 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:23.174134 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:23.174145 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:23.174675 (+ 530us) client_negotiation.cc:769] Sending connection context
0926 23:26:23.175054 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 23:26:23.175733 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27101,"thread_start_us":27080,"threads_started":1}
I20260926 23:26:23.179098 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:23.123180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41011 (local address 127.0.97.62:43199)
0926 23:26:23.148941 (+ 25761us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:23.148950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:23.151020 (+ 2070us) server_negotiation.cc:407] Connection header received
0926 23:26:23.153758 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:23.153768 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:23.153923 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:23.154091 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:23.156913 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.157970 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.161026 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.162003 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.162191 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:23.167481 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:23.167489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:23.167494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:23.167544 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:23.171207 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:23.172371 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:23.172379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:23.172383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:23.172609 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:23.173427 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:23.173433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:23.173437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:23.173875 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:23.174047 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:23.178266 (+ 4219us) server_negotiation.cc:299] Negotiation successful
0926 23:26:23.178639 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25583,"thread_start_us":25794,"threads_started":1}
I20260926 23:26:23.181995 12949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260926 23:26:23.182413 12949 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:23.183522 12949 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260926 23:26:23.186681 12736 ts_manager.cc:194] Registered new tserver with Master: b99fd6eead1740efa66d7dce953884fe (127.0.97.1:41233)
I20260926 23:26:23.189024 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:26:23.200892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:26:23 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 23:26:23.700345 12955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:23.700664 12955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:23.700872 12955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:23.708557 12955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:23.708750 12955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:23.708961 12955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:26:23.719736 12955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:41887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:23.721243 12955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:23.722600 12955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:23.736225 12969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:23.736220 12972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:23.736212 12970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:23.737562 12955 server_base.cc:1034] running on GCE node
I20260926 23:26:23.738351 12955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:23.739390 12955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:23.740705 12955 hybrid_clock.cc:648] HybridClock initialized: now 1790465183740602 us; error 71 us; skew 500 ppm
I20260926 23:26:23.741250 12955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:23 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:23.744906 12955 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:26:23.746769 12955 webserver.cc:460] Webserver started at http://127.0.97.2:33905/ using document root <none> and password file <none>
I20260926 23:26:23.747634 12955 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:23.747757 12955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:23.748193 12955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:26:23 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:23.766048 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "3f3abbac78ce420e95d9d3f986370e73"
format_stamp: "Formatted at 2026-09-26 23:26:23 on dist-test-slave-q5g8"
server_key: "d8f4b72153f66e6a6e17f524604fb21d"
server_key_iv: "6a9f184a6384623009d1c83593f482d6"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:23.767031 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3f3abbac78ce420e95d9d3f986370e73"
format_stamp: "Formatted at 2026-09-26 23:26:23 on dist-test-slave-q5g8"
server_key: "d8f4b72153f66e6a6e17f524604fb21d"
server_key_iv: "6a9f184a6384623009d1c83593f482d6"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:23.772866 12955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:26:23.788869 12979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:23.790728 12955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:26:23.790956 12955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "3f3abbac78ce420e95d9d3f986370e73"
format_stamp: "Formatted at 2026-09-26 23:26:23 on dist-test-slave-q5g8"
server_key: "d8f4b72153f66e6a6e17f524604fb21d"
server_key_iv: "6a9f184a6384623009d1c83593f482d6"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:23.791203 12955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:23.821933 12955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:23.826473 12955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:23.826870 12955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:23.827946 12955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:23.829541 12955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:23.829653 12955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:23.829814 12955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:23.829890 12955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:23.865988 12955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44221
I20260926 23:26:23.866035 13091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44221 every 8 connection(s)
I20260926 23:26:23.867404 12955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:26:23.876942 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 12955
I20260926 23:26:23.877141 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:26:23.877697 388 external_mini_cluster.cc:1428] Setting key f2de9d0b79dc4440443ddf0e4a659837
Sep 26 23:26:23 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:23.888132 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:23.870161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46959 (local address 127.0.97.62:43199)
0926 23:26:23.870632 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:23.870638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:23.871768 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 23:26:23.873616 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:23.873621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:23.873707 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:23.873812 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:23.875319 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.876107 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.878013 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.878541 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.878689 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:23.882133 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:23.882138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:23.882141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:23.882181 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:23.884904 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:23.885661 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:23.885666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:23.885672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:23.885796 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:23.886265 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:23.886276 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:23.886278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:23.886580 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:23.886741 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:23.887465 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 23:26:23.887725 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
I20260926 23:26:23.889254 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:23.870772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.2:46959)
0926 23:26:23.871572 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:23.871619 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:23.873352 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:23.874026 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:23.874034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:23.874521 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:23.875146 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:23.875177 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.876258 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.876262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:23.877855 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:23.877872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:23.878796 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:23.878804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:23.879086 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:23.879827 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:23.879860 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:23.881918 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:23.885062 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:23.885072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:23.885091 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:23.885531 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:23.885909 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:23.885915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:23.885919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:23.886143 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:23.886712 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:23.886720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:23.887086 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 23:26:23.887364 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 23:26:23.887931 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":316,"threads_started":1}
I20260926 23:26:23.890915 13092 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260926 23:26:23.891269 13092 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:23.892072 13092 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260926 23:26:23.893741 12736 ts_manager.cc:194] Registered new tserver with Master: 3f3abbac78ce420e95d9d3f986370e73 (127.0.97.2:44221)
I20260926 23:26:23.894605 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46959
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:26:23.934420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:24.193656 12949 heartbeater.cc:503] Master 127.0.97.62:43199 was elected leader, sending a full tablet report...
W20260926 23:26:24.285821 13098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:24.286186 13098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:24.286341 13098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:24.294224 13098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:24.294368 13098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:24.294572 13098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:26:24.309830 13098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:41887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:24.311264 13098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:24.312790 13098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:24.330926 13106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:24.330900 13104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:24.330832 13103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:24.331760 13098 server_base.cc:1034] running on GCE node
I20260926 23:26:24.332489 13098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:24.333545 13098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:24.334789 13098 hybrid_clock.cc:648] HybridClock initialized: now 1790465184334744 us; error 44 us; skew 500 ppm
I20260926 23:26:24.335151 13098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:24 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:24.338605 13098 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:26:24.340178 13098 webserver.cc:460] Webserver started at http://127.0.97.3:43335/ using document root <none> and password file <none>
I20260926 23:26:24.340932 13098 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:24.341041 13098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:24.341303 13098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:26:24 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:24.358618 13098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "58d7c9cd587f496eb6cc12c254a821d7"
format_stamp: "Formatted at 2026-09-26 23:26:24 on dist-test-slave-q5g8"
server_key: "2567589ed0b0369a12bc6bff7b363ec7"
server_key_iv: "16cb3d350179ab4d1a0658932e748b99"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:24.359730 13098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "58d7c9cd587f496eb6cc12c254a821d7"
format_stamp: "Formatted at 2026-09-26 23:26:24 on dist-test-slave-q5g8"
server_key: "2567589ed0b0369a12bc6bff7b363ec7"
server_key_iv: "16cb3d350179ab4d1a0658932e748b99"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:24.366109 13098 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260926 23:26:24.381697 13113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:24.383381 13098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:26:24.383564 13098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "58d7c9cd587f496eb6cc12c254a821d7"
format_stamp: "Formatted at 2026-09-26 23:26:24 on dist-test-slave-q5g8"
server_key: "2567589ed0b0369a12bc6bff7b363ec7"
server_key_iv: "16cb3d350179ab4d1a0658932e748b99"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:24.383775 13098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:24.407891 13098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:24.411729 13098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:24.411988 13098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:24.412971 13098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:24.414335 13098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:24.414420 13098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:24.414508 13098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:24.414552 13098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:24.447894 13098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39829
I20260926 23:26:24.447932 13225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39829 every 8 connection(s)
I20260926 23:26:24.449285 13098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:26:24.459496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13098
I20260926 23:26:24.459714 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:26:24.460340 388 external_mini_cluster.cc:1428] Setting key 0f4d72b4fa9a1cb0389641d5511c14ed
Sep 26 23:26:24 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:24.471567 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:24.451946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58787 (local address 127.0.97.62:43199)
0926 23:26:24.452407 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:24.452412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:24.453587 (+ 1175us) server_negotiation.cc:407] Connection header received
0926 23:26:24.455437 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:24.455444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:24.455579 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:24.455706 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:24.457445 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.458212 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.460463 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.461102 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.461274 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:24.465179 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:24.465185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:24.465188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:24.465226 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:24.468140 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.469173 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.469177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.469180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.469328 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.469839 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.469843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.469845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.470150 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:24.470281 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:24.471068 (+ 787us) server_negotiation.cc:299] Negotiation successful
0926 23:26:24.471272 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":221,"threads_started":1}
I20260926 23:26:24.472581 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:24.452553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.3:58787)
0926 23:26:24.453351 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:24.453401 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:24.455125 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:24.455972 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:24.455982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:24.456533 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:24.457226 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:24.457245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.458393 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.458397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:24.460204 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:24.460221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.461360 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.461365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:24.461701 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:24.462548 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:24.462590 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:24.464922 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:24.468381 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:24.468393 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:24.468414 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:24.469007 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:24.469483 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:24.469487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:24.469490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:24.469718 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:24.470293 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:24.470299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:24.470652 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 23:26:24.470900 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 23:26:24.471318 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":375,"threads_started":1}
I20260926 23:26:24.474196 13226 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260926 23:26:24.474543 13226 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:24.475389 13226 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260926 23:26:24.477020 12736 ts_manager.cc:194] Registered new tserver with Master: 58d7c9cd587f496eb6cc12c254a821d7 (127.0.97.3:39829)
I20260926 23:26:24.477807 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58787
I20260926 23:26:24.489039 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:26:24.490707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 12816
Sep 26 23:26:24 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:26:24.508282 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:24.491780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.0.97.1:41233)
0926 23:26:24.492449 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:24.492457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:24.492483 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:26:24.492544 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:24.492551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:24.492853 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:24.492985 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:24.494485 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.495659 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.497195 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.497757 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.497860 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:24.502126 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:24.502139 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:24.502154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:24.502196 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:24.505395 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.506099 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.506106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.506108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.506257 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.506727 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.506733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.506736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.507165 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:24.507408 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:24.507668 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 23:26:24.507880 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":342,"threads_started":1}
I20260926 23:26:24.509647 12944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:24.596946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 12816
I20260926 23:26:24.613895 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 12955
Sep 26 23:26:24 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:26:24.631474 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:24.614838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.0.97.2:44221)
0926 23:26:24.615390 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:24.615398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:24.615427 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:26:24.615489 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:24.615496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:24.615736 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:24.615926 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:24.617382 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.618500 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.620007 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.620593 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.620831 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:24.625350 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:24.625362 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:24.625377 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:24.625424 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:24.628316 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.629190 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.629198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.629200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.629354 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.629747 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.629757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.629759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.630116 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:24.630435 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:24.630802 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 23:26:24.631074 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
I20260926 23:26:24.632894 13087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:24.720140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 12955
I20260926 23:26:24.740281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13098
Sep 26 23:26:24 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:26:24.758351 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:24.741325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.0.97.3:39829)
0926 23:26:24.741958 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:24.741965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:24.741993 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:26:24.742053 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:24.742060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:24.742279 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:24.742403 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:24.743713 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.744788 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.746156 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:24.746731 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:24.746835 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:24.751814 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:24.751830 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:24.751847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:24.751888 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:24.755086 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.755955 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.755964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.755966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.756181 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:24.756632 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:24.756636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:24.756638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:24.757053 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:24.757363 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:24.757733 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 23:26:24.757950 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":323,"threads_started":1}
I20260926 23:26:24.759969 13221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:24.844834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13098
I20260926 23:26:24.861096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 12704
I20260926 23:26:24.862389 12766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:24.974455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 12704
I20260926 23:26:25.000078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43199
--webserver_interface=127.0.97.62
--webserver_port=41075
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43199
--ranger_config_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:25.345759 13238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:25.346062 13238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:25.346232 13238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:25.353487 13238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:25.353598 13238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:25.353688 13238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:25.353799 13238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:25.353884 13238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:25.364065 13238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43199
--ranger_config_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:25.365449 13238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:25.366851 13238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:25.379372 13243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:25.379442 13244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:25.379402 13246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:25.380373 13238 server_base.cc:1034] running on GCE node
I20260926 23:26:25.381299 13238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:25.382758 13238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:25.383984 13238 hybrid_clock.cc:648] HybridClock initialized: now 1790465185383944 us; error 59 us; skew 500 ppm
I20260926 23:26:25.384351 13238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:25 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:25.387460 13238 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:26:25.389089 13238 webserver.cc:460] Webserver started at http://127.0.97.62:41075/ using document root <none> and password file <none>
I20260926 23:26:25.389744 13238 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:25.389894 13238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:26:25 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:25.450641 13238 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:26:25.454694 13254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:25.456367 13238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 23:26:25.456569 13238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "36845a2d41174672891df711ac84ad19"
format_stamp: "Formatted at 2026-09-26 23:26:21 on dist-test-slave-q5g8"
server_key: "e7b818f21b5c1977b5cdc2ae8c80bf03"
server_key_iv: "522e13e8eaaf73fb1c9d3bcb254ec328"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:25.457633 13238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:25.476383 13238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:25.480230 13238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:25.480549 13238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:25.499787 13238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43199
I20260926 23:26:25.499795 13305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43199 every 8 connection(s)
I20260926 23:26:25.501494 13238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:26:25.505069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13238
I20260926 23:26:25.505890 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41233
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43199
--webserver_port=44959
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:25.509260 13306 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:26:25.513127 13306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Bootstrap starting.
I20260926 23:26:25.542867 13306 log.cc:824] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:25.555872 13306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:26:25.556723 13306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Bootstrap complete.
I20260926 23:26:25.571748 13306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:25.573441 13306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36845a2d41174672891df711ac84ad19, State: Initialized, Role: FOLLOWER
I20260926 23:26:25.574378 13306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:25.574733 13306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:26:25.574868 13306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:26:25.575073 13306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:26:25.582532 13306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:25.583320 13306 leader_election.cc:302] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36845a2d41174672891df711ac84ad19; no voters:
I20260926 23:26:25.584092 13306 leader_election.cc:288] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:26:25.584306 13310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:26:25.585676 13310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [term 2 LEADER]: Becoming Leader. State: Replica: 36845a2d41174672891df711ac84ad19, State: Running, Role: LEADER
I20260926 23:26:25.586472 13310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } }
I20260926 23:26:25.586740 13306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:25.589332 13312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36845a2d41174672891df711ac84ad19. Latest consensus state: current_term: 2 leader_uuid: "36845a2d41174672891df711ac84ad19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } } }
I20260926 23:26:25.589802 13312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:25.590520 13311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "36845a2d41174672891df711ac84ad19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36845a2d41174672891df711ac84ad19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43199 } } }
I20260926 23:26:25.590903 13311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19 [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:25.595062 13316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:26:25.602725 13316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:26:25.604388 13316 catalog_manager.cc:1259] Loaded cluster ID: dcb16e442617431dbf568ad73a22814b
I20260926 23:26:25.604616 13316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:26:25.609300 13316 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:26:25.611251 13316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Loaded TSK: 0
I20260926 23:26:25.613219 13316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:26:25.618611 13306 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 23:26:26.036222 13308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:26.036717 13308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:26.036885 13308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:26.049013 13308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:26.049206 13308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:26.049429 13308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:26:26.066272 13308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44959
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:26.068264 13308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:26.070322 13308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:26.098542 13344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:26.099012 13345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:26.104951 13347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:26.105353 13308 server_base.cc:1034] running on GCE node
I20260926 23:26:26.106379 13308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:26.107620 13308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:26.108958 13308 hybrid_clock.cc:648] HybridClock initialized: now 1790465186108884 us; error 70 us; skew 500 ppm
I20260926 23:26:26.109359 13308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:26 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:26.115218 13308 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:26:26.117563 13308 webserver.cc:460] Webserver started at http://127.0.97.1:44959/ using document root <none> and password file <none>
I20260926 23:26:26.118549 13308 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:26.118754 13308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:26:26 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:26.154676 13308 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.008s sys 0.004s
I20260926 23:26:26.159143 13355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:26.161342 13308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:26:26.161643 13308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "b99fd6eead1740efa66d7dce953884fe"
format_stamp: "Formatted at 2026-09-26 23:26:22 on dist-test-slave-q5g8"
server_key: "a5b2edc340bbfd34b3e596463690d014"
server_key_iv: "4172ee58307eac50afe02746ad3ac10a"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:26.163280 13308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:26.209440 13308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:26.215762 13308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:26.216183 13308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:26.217602 13308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:26.219679 13308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:26.219811 13308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:26.219935 13308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:26.220026 13308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:26.283850 13308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41233
I20260926 23:26:26.283903 13468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41233 every 8 connection(s)
I20260926 23:26:26.285810 13308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:26:26.290624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13308
I20260926 23:26:26.291433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44221
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43199
--webserver_port=33905
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:26:26 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:26.326617 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:26.289673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.1:54893)
0926 23:26:26.290667 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:26.290735 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:26.292948 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:26.295481 (+ 2533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:26.295493 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:26.296158 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:26.297071 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:26.297101 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:26.299346 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:26.299354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:26.302220 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:26.302245 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:26.304034 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:26.304045 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:26.304408 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:26.305400 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:26.305438 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:26.308869 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:26.314515 (+ 5646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:26.314529 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:26.314553 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:26.315428 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:26.316317 (+ 889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:26.316325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:26.316331 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:26.316650 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:26.323387 (+ 6737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:26.323398 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:26.323954 (+ 556us) client_negotiation.cc:769] Sending connection context
0926 23:26:26.324305 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 23:26:26.325152 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":2949,"threads_started":1}
I20260926 23:26:26.327448 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:26.289379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54893 (local address 127.0.97.62:43199)
0926 23:26:26.290102 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:26.290121 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:26.290906 (+ 785us) server_negotiation.cc:407] Connection header received
0926 23:26:26.293784 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:26.293799 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:26.294273 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:26.294704 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:26.297391 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:26.299106 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:26.302469 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:26.303687 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:26.303954 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:26.309338 (+ 5384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:26.309353 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:26.309379 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:26.309444 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:26.314262 (+ 4818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:26.315660 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:26.315688 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:26.315700 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:26.315969 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:26.316823 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:26.316828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:26.316832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:26.317596 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:26.317865 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:26.325559 (+ 7694us) server_negotiation.cc:299] Negotiation successful
0926 23:26:26.326099 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":287,"threads_started":1}
I20260926 23:26:26.331442 13469 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260926 23:26:26.331840 13469 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:26.332871 13469 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
W20260926 23:26:26.336875 13469 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43199 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:26:26.337169 13469 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:26.337519 13469 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260926 23:26:26.339039 13469 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:26.339426 13469 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
W20260926 23:26:26.340735 13469 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:26:26.725409 13473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:26.725782 13473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:26.725999 13473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:26.734452 13473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:26.734651 13473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:26.734850 13473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:26:26.749222 13473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33905
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:26.751205 13473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:26.752887 13473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:26.770699 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:26.770737 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:26.780194 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:26.781227 13473 server_base.cc:1034] running on GCE node
I20260926 23:26:26.782436 13473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:26.784121 13473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:26.785380 13473 hybrid_clock.cc:648] HybridClock initialized: now 1790465186785302 us; error 82 us; skew 500 ppm
I20260926 23:26:26.785811 13473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:26 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:26.789875 13473 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:26:26.792022 13473 webserver.cc:460] Webserver started at http://127.0.97.2:33905/ using document root <none> and password file <none>
I20260926 23:26:26.792954 13473 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:26.793136 13473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:26:26 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:26.818403 13473 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:26:26.822244 13498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:26.824143 13473 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:26:26.824374 13473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "3f3abbac78ce420e95d9d3f986370e73"
format_stamp: "Formatted at 2026-09-26 23:26:23 on dist-test-slave-q5g8"
server_key: "d8f4b72153f66e6a6e17f524604fb21d"
server_key_iv: "6a9f184a6384623009d1c83593f482d6"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:26.825727 13473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:26.878486 13473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:26.884613 13473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:26.884989 13473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:26.892493 13473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:26.894747 13473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:26.894893 13473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:26.895033 13473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:26.895120 13473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:26:26 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:26.956331 13473 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44221
I20260926 23:26:26.956409 13612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44221 every 8 connection(s)
I20260926 23:26:26.958444 13473 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:26:26.966434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13473
I20260926 23:26:26.967157 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39829
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43199
--webserver_port=43335
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:26:27 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:27.052997 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:26.987373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54529 (local address 127.0.97.62:43199)
0926 23:26:27.010695 (+ 23322us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:27.010702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:27.014443 (+ 3741us) server_negotiation.cc:407] Connection header received
0926 23:26:27.029276 (+ 14833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:27.029285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:27.029408 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:27.029505 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:27.033430 (+ 3925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:27.034738 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:27.036787 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:27.037466 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:27.037613 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:27.043917 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:27.043925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:27.043930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:27.043979 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:27.048293 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:27.049541 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:27.049548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:27.049552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:27.049825 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:27.050455 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:27.050462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:27.050468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:27.050974 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:27.051125 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:27.052259 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0926 23:26:27.052531 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23152,"thread_start_us":23340,"threads_started":1}
I20260926 23:26:27.056110 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:26.986623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.2:54529)
0926 23:26:27.011459 (+ 24836us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:27.011529 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:27.014247 (+ 2718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:27.030273 (+ 16026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:27.030290 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:27.031961 (+ 1671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:27.033063 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:27.033102 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:27.034964 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:27.034968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:27.036552 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:27.036575 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:27.037984 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:27.037994 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:27.038323 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:27.040766 (+ 2443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:27.040802 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:27.043540 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:27.048526 (+ 4986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:27.048537 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:27.048559 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:27.049368 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:27.050004 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:27.050009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:27.050013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:27.050311 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:27.051238 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:27.051247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:27.051734 (+ 487us) client_negotiation.cc:769] Sending connection context
0926 23:26:27.053657 (+ 1923us) client_negotiation.cc:240] Negotiation successful
0926 23:26:27.054390 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24444,"thread_start_us":33960,"threads_started":1}
I20260926 23:26:27.058883 13613 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260926 23:26:27.059300 13613 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:27.060420 13613 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260926 23:26:27.063647 13271 ts_manager.cc:194] Registered new tserver with Master: 3f3abbac78ce420e95d9d3f986370e73 (127.0.97.2:44221)
I20260926 23:26:27.066082 13271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54529
I20260926 23:26:27.340857 13469 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:27.341337 13469 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260926 23:26:27.342756 13271 ts_manager.cc:194] Registered new tserver with Master: b99fd6eead1740efa66d7dce953884fe (127.0.97.1:41233)
I20260926 23:26:27.344029 13271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54893
W20260926 23:26:27.476773 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:27.477284 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:27.477481 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:27.490213 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:27.490374 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:27.490578 13614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
Sep 26 23:26:27 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:27.507637 13614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43335
--tserver_master_addrs=127.0.97.62:43199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:27.509878 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:27.511885 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:27.577203 13623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:27.582518 13624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:27.596853 13614 server_base.cc:1034] running on GCE node
W20260926 23:26:27.598402 13626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:27.599460 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:27.603029 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:27.604359 13614 hybrid_clock.cc:648] HybridClock initialized: now 1790465187604286 us; error 76 us; skew 500 ppm
I20260926 23:26:27.604835 13614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:27 dist-test-slave-q5g8 krb5kdc[12260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:27.609522 13614 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:26:27.625466 13614 webserver.cc:460] Webserver started at http://127.0.97.3:43335/ using document root <none> and password file <none>
I20260926 23:26:27.626557 13614 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:27.626745 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:26:27 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:27.653198 13614 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:26:27.656631 13634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:27.658560 13614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:26:27.658810 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "58d7c9cd587f496eb6cc12c254a821d7"
format_stamp: "Formatted at 2026-09-26 23:26:24 on dist-test-slave-q5g8"
server_key: "2567589ed0b0369a12bc6bff7b363ec7"
server_key_iv: "16cb3d350179ab4d1a0658932e748b99"
server_key_version: "kuduclusterkey@0"
I20260926 23:26:27.659970 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:27.716768 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:27.722316 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:27.722661 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:27.730122 13614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:27.738628 13614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:27.738873 13614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:27.739043 13614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:27.739161 13614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:26:27 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:26:28.071557 13613 heartbeater.cc:503] Master 127.0.97.62:43199 was elected leader, sending a full tablet report...
I20260926 23:26:28.145560 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39829 every 8 connection(s)
I20260926 23:26:28.145975 13614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39829
I20260926 23:26:28.148335 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:26:28.150022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13614
Sep 26 23:26:28 dist-test-slave-q5g8 krb5kdc[12260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:28.207549 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.151682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.0.97.62:43199)
0926 23:26:28.182486 (+ 30804us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.182496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.182527 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:26:28.182625 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.182635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.186226 (+ 3591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.186452 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:28.188989 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.190115 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.192421 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.193114 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.197282 (+ 4168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.197565 (+ 283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:28.197572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:28.197578 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:28.197613 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:28.202595 (+ 4982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.203809 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.203816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.203820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.203995 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.204589 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.204595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.204599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.205024 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:28.206687 (+ 1663us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.206843 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.207120 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":30602,"thread_start_us":47580,"threads_started":1}
I20260926 23:26:28.221019 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.157814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58999 (local address 127.0.97.62:43199)
0926 23:26:28.187140 (+ 29326us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.187149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.187173 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:26:28.187238 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.187244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.187342 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.190102 (+ 2760us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:28.190368 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.191646 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.196325 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.197104 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.198567 (+ 1463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.208062 (+ 9495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:28.208069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:28.208075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:28.208113 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:28.211965 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.213974 (+ 2009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.213981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.213985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.214161 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.218333 (+ 4172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.218340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.218344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.218769 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:28.218921 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.220215 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.220544 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1297,"server-negotiator.queue_time_us":29146,"thread_start_us":24331,"threads_started":1}
I20260926 23:26:28.223878 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.157081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43199 (local address 127.0.97.3:58999)
0926 23:26:28.170023 (+ 12942us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:28.170082 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:28.181303 (+ 11221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:28.188007 (+ 6704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:28.188027 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:28.188888 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:28.189883 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.189920 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.193404 (+ 3484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.193411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.196005 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.196032 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.197416 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.197424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.197805 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.198872 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:28.198925 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:28.207782 (+ 8857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:28.212228 (+ 4446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:28.212241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:28.212277 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:28.213789 (+ 1512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:28.214359 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:28.214364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:28.214368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:28.214704 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:28.219067 (+ 4363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:28.219078 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:28.219695 (+ 617us) client_negotiation.cc:769] Sending connection context
0926 23:26:28.221619 (+ 1924us) client_negotiation.cc:240] Negotiation successful
0926 23:26:28.222282 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12609,"thread_start_us":12046,"threads_started":1}
I20260926 23:26:28.226248 13748 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43199
I20260926 23:26:28.226647 13748 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:28.227674 13748 heartbeater.cc:511] Master 127.0.97.62:43199 requested a full tablet report, sending...
I20260926 23:26:28.229457 13271 ts_manager.cc:194] Registered new tserver with Master: 58d7c9cd587f496eb6cc12c254a821d7 (127.0.97.3:39829)
I20260926 23:26:28.230646 13271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58999
I20260926 23:26:28.231449 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:26:28.327004 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.283091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.0.97.62:43199)
0926 23:26:28.283318 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.283325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.293925 (+ 10600us) server_negotiation.cc:407] Connection header received
0926 23:26:28.294530 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.294539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.294630 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.294706 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:28.297073 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.298202 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.300301 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.301033 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.301150 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.316310 (+ 15160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:28.316319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:28.316326 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:28.316385 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:28.320430 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.321945 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.321952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.321956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.322140 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.322636 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.322642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.322647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.323865 (+ 1218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:28.324941 (+ 1076us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.326307 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.326600 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:26:28.337024 13271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 23:26:28.347919 13469 heartbeater.cc:503] Master 127.0.97.62:43199 was elected leader, sending a full tablet report...
W20260926 23:26:28.350853 13271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:26:28.393424 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.367596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.0.97.3:39829)
0926 23:26:28.374423 (+ 6827us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.374432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.381545 (+ 7113us) server_negotiation.cc:407] Connection header received
0926 23:26:28.381660 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.381668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.382015 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.382280 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:28.385375 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.387490 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.390340 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.391569 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.391811 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.392077 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.392807 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.393001 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6628,"thread_start_us":7727,"threads_started":1}
I20260926 23:26:28.394075 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.367085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39829 (local address 127.0.0.1:36320)
0926 23:26:28.381246 (+ 14161us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:28.381268 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:28.381410 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:28.384501 (+ 3091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:28.384507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:28.384538 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:28.385063 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.385080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.387681 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.387686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.390153 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.390167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.391825 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.391832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.392068 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.392139 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 23:26:28.392294 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 23:26:28.393743 (+ 1449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14011,"thread_start_us":5890,"threads_started":1}
I20260926 23:26:28.398710 13683 tablet_service.cc:1467] Processing CreateTablet for tablet 13589d84f25e402ea5df85df647f0f24 (DEFAULT_TABLE table=test-table [id=2ea886afbb3c4c62abee00833f196cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:28.400511 13683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13589d84f25e402ea5df85df647f0f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:28.405725 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.366899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44221 (local address 127.0.0.1:33722)
0926 23:26:28.373448 (+ 6549us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:28.373513 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:28.373740 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:28.376108 (+ 2368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:28.376122 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:28.376170 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:28.376772 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.376789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.385036 (+ 8247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.385047 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.388556 (+ 3509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.388583 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.404871 (+ 16288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.404880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.405073 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.405112 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 23:26:28.405207 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 23:26:28.405357 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6218,"thread_start_us":14020,"threads_started":1}
I20260926 23:26:28.406620 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.367590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.97.2:44221)
0926 23:26:28.374953 (+ 7363us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.374963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.374994 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:26:28.375110 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.375119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.375426 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.375688 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:28.382696 (+ 7008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.384307 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.390368 (+ 6061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.391784 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.391923 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.392142 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.406079 (+ 13937us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.406303 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7149,"thread_start_us":6992,"threads_started":1}
I20260926 23:26:28.407526 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.375204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.0.97.1:41233)
0926 23:26:28.393769 (+ 18565us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.393778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.393811 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:26:28.393916 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.393924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.394217 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.394461 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:28.395887 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.399008 (+ 3121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.401968 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.405108 (+ 3140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.405258 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.405505 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.406962 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.407170 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18369,"thread_start_us":19440,"threads_started":1}
I20260926 23:26:28.408087 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.375059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41233 (local address 127.0.0.1:47014)
0926 23:26:28.389146 (+ 14087us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:28.389167 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:28.389287 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:28.395292 (+ 6005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:28.395299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:28.395330 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:28.395653 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.395665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.399199 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.399205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.401720 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.401750 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.405686 (+ 3936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.405695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.406315 (+ 620us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.406368 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 23:26:28.406464 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:26:28.406615 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13925,"thread_start_us":13738,"threads_started":1}
I20260926 23:26:28.410844 13548 tablet_service.cc:1467] Processing CreateTablet for tablet 13589d84f25e402ea5df85df647f0f24 (DEFAULT_TABLE table=test-table [id=2ea886afbb3c4c62abee00833f196cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:28.412494 13548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13589d84f25e402ea5df85df647f0f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:28.412915 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 13589d84f25e402ea5df85df647f0f24 (DEFAULT_TABLE table=test-table [id=2ea886afbb3c4c62abee00833f196cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:28.415154 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13589d84f25e402ea5df85df647f0f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:28.425001 13776 tablet_bootstrap.cc:492] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Bootstrap starting.
I20260926 23:26:28.432112 13776 tablet_bootstrap.cc:654] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:28.433787 13776 log.cc:824] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:28.436388 13777 tablet_bootstrap.cc:492] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Bootstrap starting.
I20260926 23:26:28.445516 13776 tablet_bootstrap.cc:492] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: No bootstrap required, opened a new log
I20260926 23:26:28.445928 13776 ts_tablet_manager.cc:1397] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.008s
I20260926 23:26:28.450699 13777 tablet_bootstrap.cc:654] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:28.459861 13779 tablet_bootstrap.cc:492] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Bootstrap starting.
I20260926 23:26:28.452636 13777 log.cc:824] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:28.469111 13779 tablet_bootstrap.cc:654] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:28.469269 13776 raft_consensus.cc:348] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.469841 13776 raft_consensus.cc:374] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:28.469969 13776 raft_consensus.cc:729] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d7c9cd587f496eb6cc12c254a821d7, State: Initialized, Role: FOLLOWER
I20260926 23:26:28.470876 13776 consensus_queue.cc:260] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.475553 13777 tablet_bootstrap.cc:492] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: No bootstrap required, opened a new log
I20260926 23:26:28.477628 13779 log.cc:824] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:28.478642 13777 ts_tablet_manager.cc:1397] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.003s
I20260926 23:26:28.479496 13776 ts_tablet_manager.cc:1428] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Time spent starting tablet: real 0.033s user 0.008s sys 0.014s
I20260926 23:26:28.479949 13748 heartbeater.cc:503] Master 127.0.97.62:43199 was elected leader, sending a full tablet report...
I20260926 23:26:28.490468 13777 raft_consensus.cc:348] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.491014 13777 raft_consensus.cc:374] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:28.491201 13777 raft_consensus.cc:729] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3abbac78ce420e95d9d3f986370e73, State: Initialized, Role: FOLLOWER
I20260926 23:26:28.491967 13777 consensus_queue.cc:260] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.494740 13779 tablet_bootstrap.cc:492] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: No bootstrap required, opened a new log
I20260926 23:26:28.495265 13779 ts_tablet_manager.cc:1397] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.004s
I20260926 23:26:28.514623 13777 ts_tablet_manager.cc:1428] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Time spent starting tablet: real 0.036s user 0.016s sys 0.018s
I20260926 23:26:28.527518 13779 raft_consensus.cc:348] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.528407 13779 raft_consensus.cc:374] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:28.528594 13779 raft_consensus.cc:729] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99fd6eead1740efa66d7dce953884fe, State: Initialized, Role: FOLLOWER
I20260926 23:26:28.529613 13779 consensus_queue.cc:260] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
W20260926 23:26:28.533041 13615 tablet.cc:2369] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:28.538415 13779 ts_tablet_manager.cc:1428] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Time spent starting tablet: real 0.043s user 0.021s sys 0.001s
W20260926 23:26:28.539134 13470 tablet.cc:2369] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:28.660398 13781 raft_consensus.cc:482] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:26:28.660796 13781 raft_consensus.cc:504] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.662995 13781 leader_election.cc:288] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f3abbac78ce420e95d9d3f986370e73 (127.0.97.2:44221), b99fd6eead1740efa66d7dce953884fe (127.0.97.1:41233)
W20260926 23:26:28.670215 13753 tablet.cc:2369] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:28.671854 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.663423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44221 (local address 127.0.97.3:34847)
0926 23:26:28.663891 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:28.663914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:28.664064 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:28.665146 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:28.665151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:28.665189 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:28.665401 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.665412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.666924 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.666930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.669007 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.669026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.671057 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.671064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.671274 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.671309 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:26:28.671406 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:26:28.671547 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342}
I20260926 23:26:28.672603 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.664391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34847 (local address 127.0.97.2:44221)
0926 23:26:28.664582 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.664588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.664607 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:26:28.664713 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.664718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.664804 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.664893 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:28.665574 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.666734 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.669279 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.670491 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.670622 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.670734 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.672214 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.672375 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:26:28.674223 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13589d84f25e402ea5df85df647f0f24" candidate_uuid: "58d7c9cd587f496eb6cc12c254a821d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3abbac78ce420e95d9d3f986370e73" is_pre_election: true
I20260926 23:26:28.674904 13568 raft_consensus.cc:2454] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d7c9cd587f496eb6cc12c254a821d7 in term 0.
I20260926 23:26:28.676014 13635 leader_election.cc:302] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3abbac78ce420e95d9d3f986370e73, 58d7c9cd587f496eb6cc12c254a821d7; no voters:
I20260926 23:26:28.676553 13781 raft_consensus.cc:2781] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:26:28.676760 13781 raft_consensus.cc:482] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:26:28.676877 13781 raft_consensus.cc:3037] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:28.680384 13781 raft_consensus.cc:504] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.681736 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13589d84f25e402ea5df85df647f0f24" candidate_uuid: "58d7c9cd587f496eb6cc12c254a821d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3abbac78ce420e95d9d3f986370e73"
I20260926 23:26:28.681984 13568 raft_consensus.cc:3037] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:28.685073 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.663635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41233 (local address 127.0.97.3:51213)
0926 23:26:28.677003 (+ 13368us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:28.677028 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:28.677185 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:28.678202 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:28.678208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:28.678252 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:28.678470 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.678483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.680026 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.680032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.682101 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:28.682119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.684160 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.684166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:28.684397 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.684431 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:26:28.684533 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 23:26:28.684714 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13136,"threads_started":1}
I20260926 23:26:28.685315 13568 raft_consensus.cc:2454] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d7c9cd587f496eb6cc12c254a821d7 in term 1.
I20260926 23:26:28.685873 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.664270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51213 (local address 127.0.97.1:41233)
0926 23:26:28.664471 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.664477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.677711 (+ 13234us) server_negotiation.cc:407] Connection header received
0926 23:26:28.677814 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.677821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.677922 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.678021 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:28.678675 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.679835 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.682456 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.683728 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.683898 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.684018 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.685410 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.685627 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:26:28.685853 13635 leader_election.cc:302] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3abbac78ce420e95d9d3f986370e73, 58d7c9cd587f496eb6cc12c254a821d7; no voters:
I20260926 23:26:28.687757 13424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13589d84f25e402ea5df85df647f0f24" candidate_uuid: "58d7c9cd587f496eb6cc12c254a821d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b99fd6eead1740efa66d7dce953884fe" is_pre_election: true
I20260926 23:26:28.688515 13424 raft_consensus.cc:2454] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d7c9cd587f496eb6cc12c254a821d7 in term 0.
I20260926 23:26:28.689922 13781 leader_election.cc:288] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [CANDIDATE]: Term 1 election: Requested vote from peers 3f3abbac78ce420e95d9d3f986370e73 (127.0.97.2:44221), b99fd6eead1740efa66d7dce953884fe (127.0.97.1:41233)
I20260926 23:26:28.690160 13781 raft_consensus.cc:2781] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:26:28.690297 13781 raft_consensus.cc:686] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 1 LEADER]: Becoming Leader. State: Replica: 58d7c9cd587f496eb6cc12c254a821d7, State: Running, Role: LEADER
I20260926 23:26:28.690563 13424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13589d84f25e402ea5df85df647f0f24" candidate_uuid: "58d7c9cd587f496eb6cc12c254a821d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b99fd6eead1740efa66d7dce953884fe"
I20260926 23:26:28.690789 13424 raft_consensus.cc:3037] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:28.690850 13781 consensus_queue.cc:237] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } }
I20260926 23:26:28.695075 13424 raft_consensus.cc:2454] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d7c9cd587f496eb6cc12c254a821d7 in term 1.
I20260926 23:26:28.699992 13271 catalog_manager.cc:5495] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d7c9cd587f496eb6cc12c254a821d7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "58d7c9cd587f496eb6cc12c254a821d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d7c9cd587f496eb6cc12c254a821d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20260926 23:26:28.773888 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.767397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.0.97.3:39829)
0926 23:26:28.767615 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.767620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.767719 (+ 99us) server_negotiation.cc:407] Connection header received
0926 23:26:28.767913 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.767917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.768015 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.768160 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:26:28.768601 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.769492 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.771232 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.771657 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.771784 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.772463 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:26:28.773181 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:26:28.773344 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.773530 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.773621 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:26:28.783115 13568 raft_consensus.cc:1261] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 1 FOLLOWER]: Refusing update from remote peer 58d7c9cd587f496eb6cc12c254a821d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:28.783227 13424 raft_consensus.cc:1261] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 1 FOLLOWER]: Refusing update from remote peer 58d7c9cd587f496eb6cc12c254a821d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:28.784314 13786 consensus_queue.cc:1035] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3abbac78ce420e95d9d3f986370e73" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:28.784847 13781 consensus_queue.cc:1035] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b99fd6eead1740efa66d7dce953884fe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:28.797025 13789 mvcc.cc:205] Tried to move back new op lower bound from 7333745413244145664 to 7333745412882923520. Current Snapshot: MvccSnapshot[applied={T|T < 7333745413244145664}]
I20260926 23:26:28.813556 13270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:26:28.813885 13270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:26:28.819904 13270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36845a2d41174672891df711ac84ad19: Sending DeleteTablet for 3 replicas of tablet 13589d84f25e402ea5df85df647f0f24
I20260926 23:26:28.821506 13548 tablet_service.cc:1514] Processing DeleteTablet for tablet 13589d84f25e402ea5df85df647f0f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33722
I20260926 23:26:28.821869 13683 tablet_service.cc:1514] Processing DeleteTablet for tablet 13589d84f25e402ea5df85df647f0f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36320
I20260926 23:26:28.822507 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet 13589d84f25e402ea5df85df647f0f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47014
I20260926 23:26:28.822700 13799 tablet_replica.cc:330] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: stopping tablet replica
I20260926 23:26:28.823186 13799 raft_consensus.cc:2229] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:26:28.823659 13799 raft_consensus.cc:2258] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:28.825949 13799 ts_tablet_manager.cc:1905] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:28.826442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13308
I20260926 23:26:28.835144 13799 ts_tablet_manager.cc:1918] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:28.835337 13799 log.cc:1196] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/13589d84f25e402ea5df85df647f0f24
I20260926 23:26:28.835947 13799 ts_tablet_manager.cc:1939] T 13589d84f25e402ea5df85df647f0f24 P 3f3abbac78ce420e95d9d3f986370e73: Deleting consensus metadata
I20260926 23:26:28.837046 13801 tablet_replica.cc:330] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: stopping tablet replica
I20260926 23:26:28.837558 13801 raft_consensus.cc:2229] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:26:28.838096 13801 raft_consensus.cc:2258] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:28.838639 13255 catalog_manager.cc:4841] TS 3f3abbac78ce420e95d9d3f986370e73 (127.0.97.2:44221): tablet 13589d84f25e402ea5df85df647f0f24 (table test-table [id=2ea886afbb3c4c62abee00833f196cb4]) successfully deleted
I20260926 23:26:28.841979 13800 tablet_replica.cc:330] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: stopping tablet replica
I20260926 23:26:28.842469 13800 raft_consensus.cc:2229] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:26:28.842844 13801 ts_tablet_manager.cc:1905] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:28.843150 13800 raft_consensus.cc:2258] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:28.845386 13800 ts_tablet_manager.cc:1905] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:28.852914 13800 ts_tablet_manager.cc:1918] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:28.853116 13800 log.cc:1196] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/13589d84f25e402ea5df85df647f0f24
I20260926 23:26:28.853688 13800 ts_tablet_manager.cc:1939] T 13589d84f25e402ea5df85df647f0f24 P 58d7c9cd587f496eb6cc12c254a821d7: Deleting consensus metadata
I20260926 23:26:28.854242 13801 ts_tablet_manager.cc:1918] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:28.854445 13801 log.cc:1196] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/13589d84f25e402ea5df85df647f0f24
I20260926 23:26:28.855072 13801 ts_tablet_manager.cc:1939] T 13589d84f25e402ea5df85df647f0f24 P b99fd6eead1740efa66d7dce953884fe: Deleting consensus metadata
I20260926 23:26:28.855643 13258 catalog_manager.cc:4841] TS 58d7c9cd587f496eb6cc12c254a821d7 (127.0.97.3:39829): tablet 13589d84f25e402ea5df85df647f0f24 (table test-table [id=2ea886afbb3c4c62abee00833f196cb4]) successfully deleted
I20260926 23:26:28.857132 13257 catalog_manager.cc:4841] TS b99fd6eead1740efa66d7dce953884fe (127.0.97.1:41233): tablet 13589d84f25e402ea5df85df647f0f24 (table test-table [id=2ea886afbb3c4c62abee00833f196cb4]) successfully deleted
I20260926 23:26:28.861773 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:28.827971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.0.97.1:41233)
0926 23:26:28.828176 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:28.828184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:28.833338 (+ 5154us) server_negotiation.cc:407] Connection header received
0926 23:26:28.833531 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:28.833539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:28.833652 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:28.833778 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:28.836390 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.837800 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.839858 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:28.840461 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:28.840625 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:28.852214 (+ 11589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:28.852233 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:28.852260 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:28.852317 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:28.857497 (+ 5180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.858492 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.858504 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.858510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.858697 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:28.859184 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:28.859191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:28.859196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:28.859776 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:28.860166 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:28.860955 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 23:26:28.861325 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:26:28.862936 13464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:29.030001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13308
I20260926 23:26:29.048482 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13473
I20260926 23:26:29.066818 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:29.049492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.97.2:44221)
0926 23:26:29.049696 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:29.049701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:29.049722 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:26:29.049931 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:29.049935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:29.050026 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:29.050124 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:29.052060 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:29.053024 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:29.055277 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:29.055761 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:29.055872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:29.059621 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:29.059638 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:29.059656 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:29.059696 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:29.063270 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:29.064224 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:29.064231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:29.064233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:29.064449 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:29.065042 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:29.065049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:29.065052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:29.065524 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:29.065871 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:29.066204 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 23:26:29.066444 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:26:29.067664 13608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:29.162796 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13473
I20260926 23:26:29.187083 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13614
I20260926 23:26:29.204481 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:29.187930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.0.97.3:39829)
0926 23:26:29.188128 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:29.188133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:29.188228 (+ 95us) server_negotiation.cc:407] Connection header received
0926 23:26:29.188423 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:29.188427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:29.188505 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:29.188624 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:29.190592 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:29.191437 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:29.193595 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:29.194164 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:29.194273 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:29.197854 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:29.197868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:29.197885 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:29.197940 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:29.201249 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:29.202247 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:29.202254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:29.202256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:29.202447 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:29.202912 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:29.202916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:29.202917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:29.203286 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:29.203524 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:29.203913 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 23:26:29.204106 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:26:29.205428 13743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:29.334343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13614
I20260926 23:26:29.354503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13238
I20260926 23:26:29.355769 13301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:29.472031 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13238
I20260926 23:26:29.495082 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 23:26:29.841602 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 23:26:29.841815 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 23:26:30.188323 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 23:26:30.188 UTC [12290] LOG: received smart shutdown request
2026-09-26 23:26:30.192 UTC [12290] LOG: background worker "logical replication launcher" (PID 12299) exited with exit code 1
2026-09-26 23:26:30.192 UTC [12294] LOG: shutting down
2026-09-26 23:26:30.382 UTC [12290] LOG: database system is shut down
2026-09-26T23:26:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56909 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:26:30 dist-test-slave-q5g8 krb5kdc[13815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:26:30 dist-test-slave-q5g8 krb5kdc[13815](info): set up 2 sockets
Sep 26 23:26:30 dist-test-slave-q5g8 krb5kdc[13815](info): commencing operation
krb5kdc: starting...
W20260926 23:26:31.649192 388 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.009s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:26:31 dist-test-slave-q5g8 krb5kdc[13815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:26:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:26:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:26:31.893158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44915
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44915,127.0.97.61:40053,127.0.97.60:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:32.235631 13828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:32.236007 13828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:32.236115 13828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:32.244468 13828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:32.244601 13828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:32.244655 13828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:32.244789 13828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:32.244845 13828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:32.255118 13828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44915,127.0.97.61:40053,127.0.97.60:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:32.256574 13828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:32.258037 13828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:32.270716 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:32.270833 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:32.270790 13833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:32.275578 13828 server_base.cc:1034] running on GCE node
I20260926 23:26:32.276453 13828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:32.278076 13828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:32.279311 13828 hybrid_clock.cc:648] HybridClock initialized: now 1790465192279271 us; error 50 us; skew 500 ppm
I20260926 23:26:32.279680 13828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:32 dist-test-slave-q5g8 krb5kdc[13815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:32.282963 13828 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:26:32.284538 13828 webserver.cc:460] Webserver started at http://127.0.97.62:45695/ using document root <none> and password file <none>
I20260926 23:26:32.285311 13828 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:32.285416 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:32.285682 13828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:32.288810 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "913c272d49c24cb8ac8521806d1dd96d"
format_stamp: "Formatted at 2026-09-26 23:26:32 on dist-test-slave-q5g8"
server_key: "dfb6d3dbc27e1069b6ee0e97163ad854"
server_key_iv: "07347e96a76a3e176fea3f3ce3dc6760"
server_key_version: "encryptionkey@0"
I20260926 23:26:32.289595 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "913c272d49c24cb8ac8521806d1dd96d"
format_stamp: "Formatted at 2026-09-26 23:26:32 on dist-test-slave-q5g8"
server_key: "dfb6d3dbc27e1069b6ee0e97163ad854"
server_key_iv: "07347e96a76a3e176fea3f3ce3dc6760"
server_key_version: "encryptionkey@0"
I20260926 23:26:32.294939 13828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 23:26:32.298568 13843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:32.300063 13828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:26:32.300257 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "913c272d49c24cb8ac8521806d1dd96d"
format_stamp: "Formatted at 2026-09-26 23:26:32 on dist-test-slave-q5g8"
server_key: "dfb6d3dbc27e1069b6ee0e97163ad854"
server_key_iv: "07347e96a76a3e176fea3f3ce3dc6760"
server_key_version: "encryptionkey@0"
I20260926 23:26:32.300488 13828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:32.335908 13828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:32.340503 13828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:32.340859 13828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:32.360546 13828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44915
I20260926 23:26:32.360585 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44915 every 8 connection(s)
I20260926 23:26:32.362097 13828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:26:32.365923 13895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:32.367897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13828
I20260926 23:26:32.368142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:26:32.368958 388 external_mini_cluster.cc:1428] Setting key f59cf9f1e8543a439cc424bd3c10f27e
I20260926 23:26:32.375830 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 26 23:26:32 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:32.405736 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.379844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.0.97.62:44915)
0926 23:26:32.380871 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:32.380888 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:32.381481 (+ 593us) server_negotiation.cc:407] Connection header received
0926 23:26:32.383545 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:32.383555 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:32.383870 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:32.384095 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:32.385460 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.387511 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.389149 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.390180 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.390390 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:32.396549 (+ 6159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:32.396564 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:32.396587 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:32.396641 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:32.400914 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:32.401702 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:32.401715 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:32.401721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:32.401932 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:32.402331 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:32.402339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:32.402345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:32.403106 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:32.403272 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:32.404023 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 23:26:32.404332 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":738,"thread_start_us":498,"threads_started":1}
I20260926 23:26:32.405792 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.379844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44915 (local address 127.0.0.1:35970)
0926 23:26:32.381283 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.381327 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:32.383150 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:32.384146 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:32.384152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:32.384855 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:32.385237 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:32.385256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.387707 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.387712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:32.388977 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:32.388991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.390474 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.390480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:32.390752 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:32.391520 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:32.391558 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:32.396199 (+ 4641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:32.401083 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:32.401094 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:32.401106 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:32.401537 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:32.402046 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:32.402049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:32.402051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:32.402212 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:32.403252 (+ 1040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:32.403262 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:32.403552 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 23:26:32.403854 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 23:26:32.404525 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":149,"spinlock_wait_cycles":64768,"thread_start_us":940,"threads_started":1}
I20260926 23:26:32.408842 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:26:32.410068 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.409674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43624)
0926 23:26:32.409843 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.409948 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 23:26:32.410346 13845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
I20260926 23:26:32.412038 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.411655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43626)
0926 23:26:32.411821 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.411897 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:26:32.412262 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40053: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 23:26:32.427856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40053
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44915,127.0.97.61:40053,127.0.97.60:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:32.461586 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } attempt: 1
I20260926 23:26:32.462857 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.462388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43632)
0926 23:26:32.462616 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.462725 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 23:26:32.463985 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.463574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43640)
0926 23:26:32.463729 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.463809 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:26:32.464215 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40053: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
I20260926 23:26:32.532510 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } attempt: 2
I20260926 23:26:32.533902 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.533405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43648)
0926 23:26:32.533660 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.533764 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 23:26:32.535027 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.534676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43654)
0926 23:26:32.534821 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.534890 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:26:32.535305 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40053: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
I20260926 23:26:32.626627 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } attempt: 3
I20260926 23:26:32.627883 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.627437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43662)
0926 23:26:32.627651 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.627745 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 23:26:32.628949 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.628553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43672)
0926 23:26:32.628738 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.628812 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 23:26:32.629194 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40053: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
W20260926 23:26:32.770548 13902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:32.770852 13902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:32.771041 13902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 23:26:32.772475 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } attempt: 4
I20260926 23:26:32.773708 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.773167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43674)
0926 23:26:32.773452 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.773576 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260926 23:26:32.774778 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.774396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43682)
0926 23:26:32.774548 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.774643 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 23:26:32.774986 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40053: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40053: connect: Connection refused (error 111)
W20260926 23:26:32.779160 13902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:32.779258 13902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:32.779340 13902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:32.779434 13902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:32.779517 13902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:32.789896 13902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44915,127.0.97.61:40053,127.0.97.60:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:32.791292 13902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:32.792625 13902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:32.804625 13907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:32.804801 13908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:32.804762 13910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:32.805938 13902 server_base.cc:1034] running on GCE node
I20260926 23:26:32.806775 13902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:32.808291 13902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:32.809520 13902 hybrid_clock.cc:648] HybridClock initialized: now 1790465192809479 us; error 42 us; skew 500 ppm
I20260926 23:26:32.809840 13902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:32 dist-test-slave-q5g8 krb5kdc[13815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:32.813094 13902 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 23:26:32.814536 13902 webserver.cc:460] Webserver started at http://127.0.97.61:33247/ using document root <none> and password file <none>
I20260926 23:26:32.815198 13902 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:32.815294 13902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:32.815546 13902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:32.818477 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data/instance:
uuid: "a5b28ae9efc045469eb831ac816b4b80"
format_stamp: "Formatted at 2026-09-26 23:26:32 on dist-test-slave-q5g8"
server_key: "2b3425d57f59c6aea9c8344836320720"
server_key_iv: "484a4a9b2c00e526df59f75193f4c960"
server_key_version: "encryptionkey@0"
I20260926 23:26:32.819222 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/wal/instance:
uuid: "a5b28ae9efc045469eb831ac816b4b80"
format_stamp: "Formatted at 2026-09-26 23:26:32 on dist-test-slave-q5g8"
server_key: "2b3425d57f59c6aea9c8344836320720"
server_key_iv: "484a4a9b2c00e526df59f75193f4c960"
server_key_version: "encryptionkey@0"
I20260926 23:26:32.824488 13902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 23:26:32.828294 13917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:32.829779 13902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:26:32.829977 13902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/wal
uuid: "a5b28ae9efc045469eb831ac816b4b80"
format_stamp: "Formatted at 2026-09-26 23:26:32 on dist-test-slave-q5g8"
server_key: "2b3425d57f59c6aea9c8344836320720"
server_key_iv: "484a4a9b2c00e526df59f75193f4c960"
server_key_version: "encryptionkey@0"
I20260926 23:26:32.830185 13902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:32.867458 13902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:32.872006 13902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:32.872347 13902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:32.890769 13902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40053
I20260926 23:26:32.890795 13968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40053 every 8 connection(s)
I20260926 23:26:32.892225 13902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:26:32.896061 13969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:32.901396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13902
I20260926 23:26:32.901604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-1/wal/instance
I20260926 23:26:32.902083 388 external_mini_cluster.cc:1428] Setting key 011e0fff5573ec8483e21e621c182d0a
I20260926 23:26:32.902593 13969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:26:32 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 23:26:32.927203 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.905501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.0.97.62:44915)
0926 23:26:32.905733 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:32.905741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:32.907095 (+ 1354us) server_negotiation.cc:407] Connection header received
0926 23:26:32.909792 (+ 2697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:32.909800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:32.909906 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:32.910034 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:32.912148 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.912918 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.915443 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.915890 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.915998 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:32.920832 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:32.920837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:32.920841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:32.920885 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:32.923607 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:32.924631 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:32.924635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:32.924637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:32.924837 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:32.925290 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:32.925297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:32.925302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:32.925659 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:32.925787 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:32.926552 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 23:26:32.926818 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:26:32.928334 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.906022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44915 (local address 127.0.0.1:35982)
0926 23:26:32.906881 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.906942 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:32.909494 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:32.910350 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:32.910364 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:32.911165 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:32.911954 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:32.911979 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.913093 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.913102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:32.915265 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:32.915283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.916150 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.916159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:32.916492 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:32.917296 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:32.917331 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:32.920453 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:32.923798 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:32.923810 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:32.923834 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:32.924513 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:32.924937 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:32.924941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:32.924943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:32.925173 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:32.925788 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:32.925794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:32.926206 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 23:26:32.926449 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 23:26:32.926863 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260926 23:26:32.930094 13969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:26:32 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:26:32.946761 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.930968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43684)
0926 23:26:32.931237 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.931261 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:32.931372 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:32.932761 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:32.932768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:32.933292 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:32.933507 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:32.933521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.935697 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.935705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:32.936553 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:32.936564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.937462 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.937467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:32.937728 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:32.938370 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:32.938386 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:32.940440 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:32.943945 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:32.943953 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:32.943957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:32.944456 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:32.944953 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:32.944958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:32.944961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:32.945100 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:32.945751 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:32.945755 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:32.945893 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 23:26:32.946034 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 23:26:32.946329 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":26}
I20260926 23:26:32.946795 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.931166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.0.97.61:40053)
0926 23:26:32.931929 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:32.931939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:32.931972 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:26:32.932028 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:32.932038 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:32.932300 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:32.932446 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:32.933742 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.935525 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.936734 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:32.937306 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:32.937450 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:32.940658 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:32.940670 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:32.940730 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:32.940784 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:32.943781 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:32.944600 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:32.944617 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:32.944620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:32.944790 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:32.945206 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:32.945214 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:32.945217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:32.945599 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:32.945802 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:32.946190 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 23:26:32.946436 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":574,"thread_start_us":417,"threads_started":1}
I20260926 23:26:32.949641 13969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:26:32.950846 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.950363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44104)
0926 23:26:32.950543 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.950688 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 23:26:32.951159 13919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
I20260926 23:26:32.952894 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:32.952478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44110)
0926 23:26:32.952615 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:32.952786 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 23:26:32.953218 13969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 23:26:32.956415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34845
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44915,127.0.97.61:40053,127.0.97.60:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:33.002560 13969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 1
I20260926 23:26:33.003855 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.003374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44124)
0926 23:26:33.003595 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.003708 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:26:33.005010 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.004656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44140)
0926 23:26:33.004840 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.004905 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 23:26:33.005280 13969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
I20260926 23:26:33.073537 13969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 2
I20260926 23:26:33.074285 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } attempt: 5
I20260926 23:26:33.074889 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.074306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44146)
0926 23:26:33.074555 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.074686 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 23:26:33.076102 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.075644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44158)
0926 23:26:33.075834 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.075920 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260926 23:26:33.076424 13969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:26:33.088402 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.074926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43694)
0926 23:26:33.075237 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.075258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:33.075394 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:33.075762 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:33.075767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:33.076337 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:33.076602 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.076613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.077601 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.077607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.078460 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.078471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.079213 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.079219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.079440 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.080099 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:33.080117 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:33.082626 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:33.085885 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.085891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.085895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.086312 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.086703 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.086710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.086714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.086861 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.087424 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:33.087429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:33.087599 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 23:26:33.087738 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 23:26:33.088044 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 23:26:33.088390 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.074994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.0.97.61:40053)
0926 23:26:33.075322 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.075329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.075352 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:26:33.075540 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.075544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.075618 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.075727 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.076767 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.077441 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.078608 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.079061 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.079197 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.082857 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.082862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.082866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.082900 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.085719 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.086436 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.086442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.086446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.086580 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.086971 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.086977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.086980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.087264 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.087450 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.087858 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.088069 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260926 23:26:33.089784 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:26:33.090794 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.090410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44166)
0926 23:26:33.090567 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.090643 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 23:26:33.091694 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.091363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44178)
0926 23:26:33.091518 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.091594 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:26:33.091913 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
I20260926 23:26:33.143165 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 1
I20260926 23:26:33.144526 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.143843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44184)
0926 23:26:33.144132 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.144272 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 23:26:33.145797 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.145379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44186)
0926 23:26:33.145548 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.145650 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 23:26:33.146113 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
I20260926 23:26:33.167685 13969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 3
I20260926 23:26:33.169097 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.168397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44194)
0926 23:26:33.168711 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.168849 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:26:33.170379 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.169803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44204)
0926 23:26:33.170093 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.170203 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260926 23:26:33.170698 13969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
I20260926 23:26:33.214385 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 2
I20260926 23:26:33.215598 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.215038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44214)
0926 23:26:33.215260 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.215380 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 23:26:33.216784 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.216306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44220)
0926 23:26:33.216491 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.216587 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 23:26:33.217108 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
W20260926 23:26:33.294333 13976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:33.294627 13976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:33.294788 13976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:33.302148 13976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:33.302282 13976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:33.302369 13976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:33.302474 13976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:33.302554 13976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:33.312526 13976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44915,127.0.97.61:40053,127.0.97.60:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:33.313894 13976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:33.313957 13969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 4
I20260926 23:26:33.315169 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.314604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44228)
0926 23:26:33.314868 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.314973 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 23:26:33.315543 13976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:26:33.316203 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.315825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44232)
0926 23:26:33.315989 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.316075 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 23:26:33.316468 13969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
I20260926 23:26:33.323339 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 3
I20260926 23:26:33.324399 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.323898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44238)
0926 23:26:33.324172 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.324251 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 23:26:33.325412 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.325099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44242)
0926 23:26:33.325241 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.325317 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:26:33.325644 13895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34845: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34845: connect: Connection refused (error 111)
W20260926 23:26:33.327582 13981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:33.327618 13982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:33.327653 13984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:33.329059 13976 server_base.cc:1034] running on GCE node
I20260926 23:26:33.329804 13976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:33.331164 13976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:33.332399 13976 hybrid_clock.cc:648] HybridClock initialized: now 1790465193332358 us; error 55 us; skew 500 ppm
I20260926 23:26:33.332933 13976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:33.336165 13976 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 23:26:33.337656 13976 webserver.cc:460] Webserver started at http://127.0.97.60:38071/ using document root <none> and password file <none>
I20260926 23:26:33.338376 13976 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:33.338472 13976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:33.338740 13976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:33.341667 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data/instance:
uuid: "cfbde91ddd294d87b6eced2751fd01a7"
format_stamp: "Formatted at 2026-09-26 23:26:33 on dist-test-slave-q5g8"
server_key: "a3baeeb327e3ca7f08a48c3f1d1a5fd9"
server_key_iv: "40f893935782fd148b033013c6d10ecb"
server_key_version: "encryptionkey@0"
I20260926 23:26:33.342384 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/wal/instance:
uuid: "cfbde91ddd294d87b6eced2751fd01a7"
format_stamp: "Formatted at 2026-09-26 23:26:33 on dist-test-slave-q5g8"
server_key: "a3baeeb327e3ca7f08a48c3f1d1a5fd9"
server_key_iv: "40f893935782fd148b033013c6d10ecb"
server_key_version: "encryptionkey@0"
I20260926 23:26:33.347493 13976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:26:33.350884 13991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:33.352211 13976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 23:26:33.352344 13976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/wal
uuid: "cfbde91ddd294d87b6eced2751fd01a7"
format_stamp: "Formatted at 2026-09-26 23:26:33 on dist-test-slave-q5g8"
server_key: "a3baeeb327e3ca7f08a48c3f1d1a5fd9"
server_key_iv: "40f893935782fd148b033013c6d10ecb"
server_key_version: "encryptionkey@0"
I20260926 23:26:33.352507 13976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:33.375595 13976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:33.380067 13976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:33.380306 13976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:33.399120 13976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34845
I20260926 23:26:33.399152 14042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34845 every 8 connection(s)
I20260926 23:26:33.400615 13976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:26:33.404385 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:33.409497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 13976
I20260926 23:26:33.409684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790464978839002-388-0/minicluster-data/master-2/wal/instance
I20260926 23:26:33.410157 388 external_mini_cluster.cc:1428] Setting key 8990c4990dc9e055228ea615373075f3
I20260926 23:26:33.411243 14043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:26:33.431945 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.414029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.0.97.62:44915)
0926 23:26:33.414234 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.414240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.415054 (+ 814us) server_negotiation.cc:407] Connection header received
0926 23:26:33.415455 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.415464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.415547 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.415641 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.417399 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.418518 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.420618 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.421238 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.421367 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.425469 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.425474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.425478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.425511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.428297 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.429270 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.429274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.429278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.429411 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.429877 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.429882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.429886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.430242 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.430404 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.431303 (+ 899us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.431538 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:26:33.433166 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.412116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.0.97.60:34845)
0926 23:26:33.413012 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.413023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.413064 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:26:33.414978 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.414999 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.415347 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.415563 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.417403 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.418997 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.420958 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.421726 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.421824 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.426278 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.426287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.426299 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.426326 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.429547 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.430377 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.430385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.430388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.430532 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.430879 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.430887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.430890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.431303 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.431565 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.431899 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.432115 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":417,"threads_started":1}
I20260926 23:26:33.433185 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.414059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44915 (local address 127.0.0.1:35998)
0926 23:26:33.414891 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.414931 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:33.415241 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:33.415744 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:33.415754 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:33.416621 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:33.417214 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.417238 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.418685 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.418689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.420454 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.420470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.421452 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.421457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.421628 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.422367 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:33.422403 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:33.425247 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:33.428473 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.428483 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.428495 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.429148 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.429521 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.429526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.429530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.429754 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.430373 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:33.430380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:33.430800 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 23:26:33.431195 (+ 395us) client_negotiation.cc:240] Negotiation successful
0926 23:26:33.431674 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"spinlock_wait_cycles":22400,"thread_start_us":427,"threads_started":1}
I20260926 23:26:33.434875 14043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:33.448411 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.435649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.0.97.61:40053)
0926 23:26:33.435811 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.435815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.435831 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:26:33.435959 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.435962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.436020 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.436132 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.437226 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.437909 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.439066 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.439485 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.439614 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.442683 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.442697 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.442700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.442732 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.445688 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.446479 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.446484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.446486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.446651 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.447081 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.447085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.447088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.447426 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.447545 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.447962 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.448141 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 23:26:33.448557 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.435564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40053 (local address 127.0.0.1:43696)
0926 23:26:33.435732 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.435751 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:33.435850 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:33.436475 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:33.436483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:33.436923 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:33.437059 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.437067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.438053 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.438059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.438922 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.438932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.439629 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.439635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.439866 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.440454 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:33.440470 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:33.442491 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:33.445820 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.445827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.445832 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.446347 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.446767 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.446772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.446776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.446950 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.447548 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:33.447553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:33.447718 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 23:26:33.447887 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 23:26:33.448169 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 23:26:33.449621 14043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:26:33.452210 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.436777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.97.62:44915)
0926 23:26:33.436942 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.436946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.436963 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:26:33.437153 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.437157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.437211 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.437316 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.438488 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.439212 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.440453 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.440941 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.441067 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.445419 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.445424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.445427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.445459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.448910 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.449816 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.449824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.449828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.449956 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.450713 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.450719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.450722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.451109 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.451330 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.451641 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.451887 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:26:33.464416 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.450540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.0.97.60:34845)
0926 23:26:33.450696 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.450702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.450719 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:26:33.450766 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.450773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.450844 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.450938 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.451759 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.452648 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.453930 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.454470 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.454582 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.458326 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.458332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.458337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.458374 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.461929 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.462555 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.462560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.462563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.462736 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.463082 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.463089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.463093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.463365 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.463521 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.463999 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.464181 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:26:33.464797 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.450370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44258)
0926 23:26:33.450554 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.450581 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:33.450683 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:33.450971 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:33.450974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:33.451418 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:33.451624 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.451637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.452813 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.452819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.453801 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.453812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.454962 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.454967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.455118 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.455736 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:33.455760 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:33.458135 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:33.462074 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.462078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.462081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.462435 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.462857 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.462860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.462862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.462978 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.463497 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:33.463504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:33.463671 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 23:26:33.463921 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 23:26:33.464357 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":108}
I20260926 23:26:33.469775 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7: Bootstrap starting.
I20260926 23:26:33.470404 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.454483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.0.97.61:40053)
0926 23:26:33.454651 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.454655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.454669 (+ 14us) server_negotiation.cc:407] Connection header received
0926 23:26:33.454711 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.454714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.454765 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.454836 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.456207 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.457276 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.458885 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.459315 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.459411 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.464621 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.464629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.464634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.464715 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.467680 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.468371 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.468377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.468381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.468534 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.468981 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.468986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.468989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.469285 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.469567 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.469865 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.470093 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:26:33.473541 14043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:33.474784 14043 log.cc:824] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:33.475442 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:26:33.477727 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7: No bootstrap required, opened a new log
I20260926 23:26:33.488853 14043 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.489336 14043 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:33.489465 14043 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfbde91ddd294d87b6eced2751fd01a7, State: Initialized, Role: FOLLOWER
I20260926 23:26:33.490329 14043 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.497458 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.497933 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:26:33.500013 14043 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:33.502563 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.480558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.0.97.62:44915)
0926 23:26:33.481054 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.481061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.482280 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 23:26:33.482476 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.482484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.482566 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.482673 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.485382 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.486518 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.489238 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.489904 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.489995 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.494742 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.494749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.494754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.494799 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.499122 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.500080 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.500086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.500089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.500258 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.500710 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.500717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.500720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.501035 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.501273 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.501949 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.502197 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260926 23:26:33.502568 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.480851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.97.61:40053)
0926 23:26:33.481098 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.481108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.482277 (+ 1169us) server_negotiation.cc:407] Connection header received
0926 23:26:33.482346 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.482352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.482436 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.482534 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.485381 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.486522 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.489238 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.489893 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.489995 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.494742 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.494749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.494754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.494799 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.498671 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.499894 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.499900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.499903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.500180 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.500633 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.500638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.500641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.501035 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.501273 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.501953 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.502203 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 23:26:33.502856 13895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 4
I20260926 23:26:33.505748 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.482172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.0.97.60:34845)
0926 23:26:33.482356 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.482362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.482905 (+ 543us) server_negotiation.cc:407] Connection header received
0926 23:26:33.483182 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.483186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.483279 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.483359 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.485658 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.486804 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.489379 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.490074 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.490178 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.495105 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.495115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.495123 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.495178 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.499267 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.500310 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.500316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.500321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.500496 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.500982 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.500989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.500993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.501417 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.501735 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.502444 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.505155 (+ 2711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:26:33.520784 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.503489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44276)
0926 23:26:33.503675 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.503694 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:33.503791 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:33.504837 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:33.504842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:33.505283 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:33.505498 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.505512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.507447 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.507455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.509102 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.509122 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.509883 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.509889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.510079 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.510933 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:33.510967 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:33.513986 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:33.517694 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.517701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.517705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.518296 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.518760 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.518765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.518768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.518956 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.519675 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:33.519679 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:33.519870 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 23:26:33.520004 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 23:26:33.520330 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 23:26:33.522048 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.503835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.0.97.60:34845)
0926 23:26:33.504478 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.504483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.504504 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:26:33.504551 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.504555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.504619 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.505669 (+ 1050us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.506026 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.507288 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.509257 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.509706 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.509867 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.514184 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.514190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.514195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.514235 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.517548 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.518449 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.518455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.518459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.518617 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.519129 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.519133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.519136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.519528 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.521200 (+ 1672us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.521344 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.521587 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":333,"threads_started":1}
I20260926 23:26:33.526944 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: Bootstrap starting.
W20260926 23:26:33.527627 14068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:26:33.527751 14068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:26:33.531076 13895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:33.519934 13976 master_runner.cc:186] Error getting master registration for 127.0.97.62:44915: OK, instance_id { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" instance_seqno: 1790465192354869 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:26:33.531502 13976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:26:33.532371 13895 log.cc:824] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:33.535182 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: No bootstrap required, opened a new log
I20260926 23:26:33.542958 13895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.543262 13895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:33.543397 13895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913c272d49c24cb8ac8521806d1dd96d, State: Initialized, Role: FOLLOWER
I20260926 23:26:33.543955 13895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.544889 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.545481 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:26:33.546065 13895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:26:33.556196 14082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:26:33.558301 14082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:26:33.555524 13828 master_runner.cc:186] Error getting master registration for 127.0.97.61:40053: OK, instance_id { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" instance_seqno: 1790465192885510 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:26:33.563844 13828 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 23:26:33.593891 13849 tablet.cc:2369] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:33.615731 13969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } attempt: 5
Sep 26 23:26:33 dist-test-slave-q5g8 krb5kdc[13815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:26:33.630012 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.616342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34845 (local address 127.0.0.1:44282)
0926 23:26:33.616581 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:33.616600 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:33.616777 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:33.617136 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:33.617141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:33.617557 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:33.617713 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.617722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.618717 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.618724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.619644 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:33.619654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.620395 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.620401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:33.620587 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.621252 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:33.621269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:33.623279 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:33.627026 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.627031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.627035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.627541 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.628009 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:33.628016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:33.628021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:33.628181 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:33.628982 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:33.628990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:33.629149 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 23:26:33.629333 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 23:26:33.629607 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 23:26:33.630897 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:33.616382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.0.97.60:34845)
0926 23:26:33.616550 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:33.616556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:33.616711 (+ 155us) server_negotiation.cc:407] Connection header received
0926 23:26:33.616902 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:33.616907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:33.616993 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:33.617097 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:33.617874 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.618583 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.619788 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:33.620247 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:33.620373 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:33.623461 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:33.623466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:33.623469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:33.623499 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:33.626865 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.627681 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.627689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.627694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.627873 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:33.628319 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:33.628325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:33.628331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:33.628767 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:33.628918 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:33.630345 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0926 23:26:33.630584 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 23:26:33.632418 13997 tablet.cc:2369] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:33.635775 13969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: Bootstrap starting.
I20260926 23:26:33.638813 13969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:33.639834 13969 log.cc:824] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:33.642460 13969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: No bootstrap required, opened a new log
I20260926 23:26:33.652040 13969 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.652381 13969 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:33.652531 13969 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b28ae9efc045469eb831ac816b4b80, State: Initialized, Role: FOLLOWER
I20260926 23:26:33.653184 13969 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.654132 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.654577 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:26:33.655459 13969 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:33.679857 13902 master_runner.cc:263] No leader master found from master a5b28ae9efc045469eb831ac816b4b80
I20260926 23:26:33.680025 13902 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 23:26:33.684904 14096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:26:33.685148 14096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:26:33.815390 14056 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:26:33.815773 14056 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.817519 14056 leader_election.cc:288] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 913c272d49c24cb8ac8521806d1dd96d (127.0.97.62:44915), a5b28ae9efc045469eb831ac816b4b80 (127.0.97.61:40053)
I20260926 23:26:33.819039 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfbde91ddd294d87b6eced2751fd01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913c272d49c24cb8ac8521806d1dd96d" is_pre_election: true
I20260926 23:26:33.819677 13870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfbde91ddd294d87b6eced2751fd01a7 in term 0.
I20260926 23:26:33.820500 13994 leader_election.cc:302] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913c272d49c24cb8ac8521806d1dd96d, cfbde91ddd294d87b6eced2751fd01a7; no voters:
I20260926 23:26:33.821739 13944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfbde91ddd294d87b6eced2751fd01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b28ae9efc045469eb831ac816b4b80" is_pre_election: true
I20260926 23:26:33.822307 13944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfbde91ddd294d87b6eced2751fd01a7 in term 0.
I20260926 23:26:33.823160 14056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:26:33.823319 14056 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:26:33.823426 14056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:33.826926 14056 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.832659 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfbde91ddd294d87b6eced2751fd01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913c272d49c24cb8ac8521806d1dd96d"
I20260926 23:26:33.832916 13870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:33.836241 13870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfbde91ddd294d87b6eced2751fd01a7 in term 1.
I20260926 23:26:33.836759 13994 leader_election.cc:302] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913c272d49c24cb8ac8521806d1dd96d, cfbde91ddd294d87b6eced2751fd01a7; no voters:
I20260926 23:26:33.840541 13944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfbde91ddd294d87b6eced2751fd01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b28ae9efc045469eb831ac816b4b80"
I20260926 23:26:33.840844 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:33.844230 13944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfbde91ddd294d87b6eced2751fd01a7 in term 1.
I20260926 23:26:33.845172 14056 leader_election.cc:288] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [CANDIDATE]: Term 1 election: Requested vote from peers 913c272d49c24cb8ac8521806d1dd96d (127.0.97.62:44915), a5b28ae9efc045469eb831ac816b4b80 (127.0.97.61:40053)
I20260926 23:26:33.845570 14056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:26:33.846009 14056 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [term 1 LEADER]: Becoming Leader. State: Replica: cfbde91ddd294d87b6eced2751fd01a7, State: Running, Role: LEADER
I20260926 23:26:33.847304 14056 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } }
I20260926 23:26:33.863330 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfbde91ddd294d87b6eced2751fd01a7. Latest consensus state: current_term: 1 leader_uuid: "cfbde91ddd294d87b6eced2751fd01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.863859 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:33.869185 14100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:26:33.871552 14100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 23:26:33.874979 13923 tablet.cc:2369] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:33.887770 13944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [term 1 FOLLOWER]: Refusing update from remote peer cfbde91ddd294d87b6eced2751fd01a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:33.892231 14098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:33.901450 13870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [term 1 FOLLOWER]: Refusing update from remote peer cfbde91ddd294d87b6eced2751fd01a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:33.902367 14098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:33.916568 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfbde91ddd294d87b6eced2751fd01a7. Latest consensus state: current_term: 1 leader_uuid: "cfbde91ddd294d87b6eced2751fd01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.916895 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:26:33.919440 14101 mvcc.cc:205] Tried to move back new op lower bound from 7333745434149089280 to 7333745434008453120. Current Snapshot: MvccSnapshot[applied={T|T < 7333745434149089280}]
I20260926 23:26:33.927800 14102 mvcc.cc:205] Tried to move back new op lower bound from 7333745434149089280 to 7333745434008453120. Current Snapshot: MvccSnapshot[applied={T|T < 7333745434149089280}]
I20260926 23:26:33.929270 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfbde91ddd294d87b6eced2751fd01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.929525 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:26:33.934019 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfbde91ddd294d87b6eced2751fd01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.934365 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:33.941455 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfbde91ddd294d87b6eced2751fd01a7. Latest consensus state: current_term: 1 leader_uuid: "cfbde91ddd294d87b6eced2751fd01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.941756 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:26:33.949746 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfbde91ddd294d87b6eced2751fd01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.949996 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:26:33.949296 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfbde91ddd294d87b6eced2751fd01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913c272d49c24cb8ac8521806d1dd96d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44915 } } peers { permanent_uuid: "a5b28ae9efc045469eb831ac816b4b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40053 } } peers { permanent_uuid: "cfbde91ddd294d87b6eced2751fd01a7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34845 } } }
I20260926 23:26:33.950979 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:33.956063 14100 catalog_manager.cc:1347] Generated new cluster ID: 2dcf19c971154f1594b27b0ee7c46d26
I20260926 23:26:33.956188 14100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:26:33.992458 14100 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:26:33.994479 14100 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:26:34.014652 14100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfbde91ddd294d87b6eced2751fd01a7: Generated new TSK 0
I20260926 23:26:34.015889 14100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:26:34.560302 14082 catalog_manager.cc:1259] Loaded cluster ID: 2dcf19c971154f1594b27b0ee7c46d26
I20260926 23:26:34.560400 14082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: loading cluster ID for follower catalog manager: success
I20260926 23:26:34.562994 14082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: acquiring CA information for follower catalog manager: success
I20260926 23:26:34.564670 14082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 913c272d49c24cb8ac8521806d1dd96d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:26:34.687034 14096 catalog_manager.cc:1259] Loaded cluster ID: 2dcf19c971154f1594b27b0ee7c46d26
I20260926 23:26:34.687134 14096 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: loading cluster ID for follower catalog manager: success
I20260926 23:26:34.689533 14096 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: acquiring CA information for follower catalog manager: success
I20260926 23:26:34.691206 14096 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5b28ae9efc045469eb831ac816b4b80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:26:35.134984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13828
I20260926 23:26:35.136333 13890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:35.244343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13828
I20260926 23:26:35.266191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13902
I20260926 23:26:35.267371 13964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:35.373051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13902
I20260926 23:26:35.393081 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 13976
I20260926 23:26:35.394162 14038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:35.501160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 13976
2026-09-26T23:26:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4983 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:26:35 dist-test-slave-q5g8 krb5kdc[14119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:26:35 dist-test-slave-q5g8 krb5kdc[14119](info): set up 2 sockets
Sep 26 23:26:35 dist-test-slave-q5g8 krb5kdc[14119](info): commencing operation
krb5kdc: starting...
W20260926 23:26:36.649279 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:26:36 dist-test-slave-q5g8 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:26:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:26:36.905257 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34865
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:37.272828 14132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:37.273190 14132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:37.273399 14132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:37.281111 14132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:37.281240 14132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:37.281365 14132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:37.281472 14132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:37.281553 14132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:37.292508 14132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:37.294116 14132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:37.295693 14132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:37.308951 14137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:37.308916 14138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:37.308910 14140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:37.310317 14132 server_base.cc:1034] running on GCE node
I20260926 23:26:37.311146 14132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:37.312896 14132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:37.314141 14132 hybrid_clock.cc:648] HybridClock initialized: now 1790465197314083 us; error 38 us; skew 500 ppm
I20260926 23:26:37.314513 14132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:37 dist-test-slave-q5g8 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:37.318185 14132 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:26:37.319947 14132 webserver.cc:460] Webserver started at http://127.0.97.62:39239/ using document root <none> and password file <none>
I20260926 23:26:37.320813 14132 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:37.320956 14132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:37.321324 14132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:37.324512 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "49f33431f02649bc9017ae5bdff2da3c"
format_stamp: "Formatted at 2026-09-26 23:26:37 on dist-test-slave-q5g8"
server_key: "e9cd6fdc6fbab15b9d4ff40ae334f5f0"
server_key_iv: "af476ba908e2fe2e64bb5f5b7076de2b"
server_key_version: "encryptionkey@0"
I20260926 23:26:37.325402 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "49f33431f02649bc9017ae5bdff2da3c"
format_stamp: "Formatted at 2026-09-26 23:26:37 on dist-test-slave-q5g8"
server_key: "e9cd6fdc6fbab15b9d4ff40ae334f5f0"
server_key_iv: "af476ba908e2fe2e64bb5f5b7076de2b"
server_key_version: "encryptionkey@0"
I20260926 23:26:37.331009 14132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:26:37.334679 14147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:37.336146 14132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:26:37.336339 14132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "49f33431f02649bc9017ae5bdff2da3c"
format_stamp: "Formatted at 2026-09-26 23:26:37 on dist-test-slave-q5g8"
server_key: "e9cd6fdc6fbab15b9d4ff40ae334f5f0"
server_key_iv: "af476ba908e2fe2e64bb5f5b7076de2b"
server_key_version: "encryptionkey@0"
I20260926 23:26:37.336552 14132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:37.352653 14132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:37.357327 14132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:37.357614 14132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:37.377980 14132 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34865
I20260926 23:26:37.378042 14198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34865 every 8 connection(s)
I20260926 23:26:37.379679 14132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:26:37.381500 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 14132
I20260926 23:26:37.381747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:26:37.382460 388 external_mini_cluster.cc:1428] Setting key c3e745f645909b71b765de20c91edfda
I20260926 23:26:37.384266 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:37.396551 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c: Bootstrap starting.
Sep 26 23:26:37 dist-test-slave-q5g8 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:37.400460 14199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:37.401731 14199 log.cc:824] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:37.405653 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c: No bootstrap required, opened a new log
I20260926 23:26:37.407513 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:37.384474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.0.97.62:34865)
0926 23:26:37.385315 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:37.385330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:37.385376 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:26:37.387220 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:37.387245 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:37.387711 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:37.388174 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:37.389868 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:37.391295 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:37.392748 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:37.393536 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:37.393818 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:37.398317 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:37.398327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:37.398350 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:37.398398 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:37.401946 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:37.403068 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:37.403076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:37.403084 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:37.403348 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:37.403935 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:37.403940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:37.403943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:37.404738 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:37.405060 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:37.405503 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 23:26:37.406007 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":389,"threads_started":1}
I20260926 23:26:37.412178 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:26:37.415496 14199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f33431f02649bc9017ae5bdff2da3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260926 23:26:37.416021 14199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:37.416159 14199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f33431f02649bc9017ae5bdff2da3c, State: Initialized, Role: FOLLOWER
I20260926 23:26:37.416875 14199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f33431f02649bc9017ae5bdff2da3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260926 23:26:37.417101 14199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:26:37.417255 14199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:26:37.417456 14199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:37.419757 14199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f33431f02649bc9017ae5bdff2da3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260926 23:26:37.420341 14199 leader_election.cc:302] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f33431f02649bc9017ae5bdff2da3c; no voters:
I20260926 23:26:37.422320 14199 leader_election.cc:288] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:26:37.422498 14204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:26:37.423308 14204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [term 1 LEADER]: Becoming Leader. State: Replica: 49f33431f02649bc9017ae5bdff2da3c, State: Running, Role: LEADER
I20260926 23:26:37.423939 14204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f33431f02649bc9017ae5bdff2da3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } }
I20260926 23:26:37.424614 14199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:37.427361 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f33431f02649bc9017ae5bdff2da3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f33431f02649bc9017ae5bdff2da3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } } }
I20260926 23:26:37.428395 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:37.427441 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f33431f02649bc9017ae5bdff2da3c. Latest consensus state: current_term: 1 leader_uuid: "49f33431f02649bc9017ae5bdff2da3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f33431f02649bc9017ae5bdff2da3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34865 } } }
I20260926 23:26:37.428655 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:37.430285 14213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:26:37.436542 14213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:26:37.447135 14213 catalog_manager.cc:1347] Generated new cluster ID: c1ad61998e7f48afbf0b19cacf3496ec
I20260926 23:26:37.447309 14213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:26:37.462810 14213 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:26:37.464386 14213 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:26:37.475730 14213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f33431f02649bc9017ae5bdff2da3c: Generated new TSK 0
I20260926 23:26:37.476851 14213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:26:37.526419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 14132
I20260926 23:26:37.527815 14194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:37.627719 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 14132
2026-09-26T23:26:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2123 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:26:37 dist-test-slave-q5g8 krb5kdc[14227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:26:37 dist-test-slave-q5g8 krb5kdc[14227](info): set up 2 sockets
Sep 26 23:26:37 dist-test-slave-q5g8 krb5kdc[14227](info): commencing operation
krb5kdc: starting...
W20260926 23:26:38.753252 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:26:38 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:26:39.018710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37639
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:39.374702 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:39.375023 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:39.375213 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:39.382848 14240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:39.382975 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:39.383070 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:39.383157 14240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:39.383225 14240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:39.394181 14240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:39.395701 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:39.397174 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:39.409945 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:39.409910 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:39.410024 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:39.411343 14240 server_base.cc:1034] running on GCE node
I20260926 23:26:39.412240 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:39.413822 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:39.415103 14240 hybrid_clock.cc:648] HybridClock initialized: now 1790465199415053 us; error 54 us; skew 500 ppm
I20260926 23:26:39.415532 14240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:39 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:39.424733 14240 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:26:39.426430 14240 webserver.cc:460] Webserver started at http://127.0.97.62:35085/ using document root <none> and password file <none>
I20260926 23:26:39.427153 14240 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:39.427307 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:39.427634 14240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:39.430833 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "935681ae62ee43a296022f52baa843ef"
format_stamp: "Formatted at 2026-09-26 23:26:39 on dist-test-slave-q5g8"
server_key: "f4e99196422e1ce87668e0fe99720654"
server_key_iv: "b26a84b830da5615c9e506bb2d660405"
server_key_version: "encryptionkey@0"
I20260926 23:26:39.431695 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "935681ae62ee43a296022f52baa843ef"
format_stamp: "Formatted at 2026-09-26 23:26:39 on dist-test-slave-q5g8"
server_key: "f4e99196422e1ce87668e0fe99720654"
server_key_iv: "b26a84b830da5615c9e506bb2d660405"
server_key_version: "encryptionkey@0"
I20260926 23:26:39.437252 14240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 23:26:39.440970 14255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:39.442576 14240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 23:26:39.442768 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "935681ae62ee43a296022f52baa843ef"
format_stamp: "Formatted at 2026-09-26 23:26:39 on dist-test-slave-q5g8"
server_key: "f4e99196422e1ce87668e0fe99720654"
server_key_iv: "b26a84b830da5615c9e506bb2d660405"
server_key_version: "encryptionkey@0"
I20260926 23:26:39.442991 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:39.465920 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:39.470818 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:39.471139 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:39.491531 14240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37639
I20260926 23:26:39.491559 14306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37639 every 8 connection(s)
I20260926 23:26:39.493435 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:26:39.498292 14307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:39.503572 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 14240
I20260926 23:26:39.503846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:26:39.504769 388 external_mini_cluster.cc:1428] Setting key dec3bbbc680436c25c42cad4b3582c7e
I20260926 23:26:39.510304 14307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Bootstrap starting.
I20260926 23:26:39.514209 14307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:39.515556 14307 log.cc:824] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:39.518694 14307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: No bootstrap required, opened a new log
Sep 26 23:26:39 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:39.528286 14307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:39.528888 14307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:39.529017 14307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935681ae62ee43a296022f52baa843ef, State: Initialized, Role: FOLLOWER
I20260926 23:26:39.529763 14307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:39.530007 14307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:26:39.530180 14307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:26:39.530352 14307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:39.530558 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:39.507195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.0.97.62:37639)
0926 23:26:39.508206 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:39.508223 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:39.508278 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:26:39.510201 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:39.510227 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:39.510740 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:39.511115 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:39.513108 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:39.514531 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:39.516440 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:39.517231 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:39.517437 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:39.522131 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:39.522139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:39.522158 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:39.522203 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:39.525787 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:39.526721 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:39.526729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:39.526737 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:39.526926 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:39.527356 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:39.527359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:39.527361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:39.527928 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:39.528146 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:39.528759 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 23:26:39.529297 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":505,"threads_started":1}
I20260926 23:26:39.533342 14307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:39.533942 14307 leader_election.cc:302] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 935681ae62ee43a296022f52baa843ef; no voters:
I20260926 23:26:39.534606 14307 leader_election.cc:288] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:26:39.534744 14312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:26:39.535466 14312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 LEADER]: Becoming Leader. State: Replica: 935681ae62ee43a296022f52baa843ef, State: Running, Role: LEADER
I20260926 23:26:39.536036 14312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:39.536854 14307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:39.538781 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "935681ae62ee43a296022f52baa843ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } } }
I20260926 23:26:39.539122 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:39.538875 14314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 935681ae62ee43a296022f52baa843ef. Latest consensus state: current_term: 1 leader_uuid: "935681ae62ee43a296022f52baa843ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } } }
I20260926 23:26:39.539330 14314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:39.539983 14319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:26:39.547566 14319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:26:39.558296 14319 catalog_manager.cc:1347] Generated new cluster ID: 9e3c92d4899641b88ac312eeac4eb34f
I20260926 23:26:39.558426 14319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:26:39.595628 14319 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:26:39.597707 14319 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:26:39.615046 14319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Generated new TSK 0
I20260926 23:26:39.616072 14319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:26:39.688658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37639
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:40.049682 14333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:40.050007 14333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:40.050107 14333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:40.058203 14333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:40.058352 14333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:40.058480 14333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:26:40.069206 14333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:40.070636 14333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:40.072006 14333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:40.086989 14338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:40.087038 14339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:40.086954 14341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:40.093537 14333 server_base.cc:1034] running on GCE node
I20260926 23:26:40.094234 14333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:40.095283 14333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:40.096503 14333 hybrid_clock.cc:648] HybridClock initialized: now 1790465200096428 us; error 63 us; skew 500 ppm
I20260926 23:26:40.096911 14333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:40 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:40.100231 14333 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:26:40.101940 14333 webserver.cc:460] Webserver started at http://127.0.97.1:45061/ using document root <none> and password file <none>
I20260926 23:26:40.102726 14333 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:40.102880 14333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:40.103196 14333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:40.106364 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "26366434d62d4acda0642a34a218a82e"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "0d487b342a54c1c79fd78d12274d474c"
server_key_iv: "3f4f6c136eac6316ab28f03c856cf146"
server_key_version: "encryptionkey@0"
I20260926 23:26:40.107244 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "26366434d62d4acda0642a34a218a82e"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "0d487b342a54c1c79fd78d12274d474c"
server_key_iv: "3f4f6c136eac6316ab28f03c856cf146"
server_key_version: "encryptionkey@0"
I20260926 23:26:40.113821 14333 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 23:26:40.118191 14348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:40.119917 14333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:26:40.120146 14333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "26366434d62d4acda0642a34a218a82e"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "0d487b342a54c1c79fd78d12274d474c"
server_key_iv: "3f4f6c136eac6316ab28f03c856cf146"
server_key_version: "encryptionkey@0"
I20260926 23:26:40.120411 14333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:40.141657 14333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:40.146549 14333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:40.146857 14333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:40.147955 14333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:40.149451 14333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:40.149569 14333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:40.149724 14333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:40.149794 14333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:40.185264 14333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41807
I20260926 23:26:40.185328 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41807 every 8 connection(s)
I20260926 23:26:40.186686 14333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:26:40.194096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 14333
I20260926 23:26:40.194344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:26:40.194981 388 external_mini_cluster.cc:1428] Setting key 2762511e007eebedb5fda7380d676d66
Sep 26 23:26:40 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:40.210031 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:40.189608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55071 (local address 127.0.97.62:37639)
0926 23:26:40.190128 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:40.190134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:40.191297 (+ 1163us) server_negotiation.cc:407] Connection header received
0926 23:26:40.193289 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:40.193294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:40.193367 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:40.193511 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:40.195356 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.196333 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.198275 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.199079 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.199241 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:40.202901 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:40.202911 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:40.202919 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:40.202968 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:40.206178 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:40.207288 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:40.207293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:40.207295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:40.207433 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:40.208012 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:40.208016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:40.208018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:40.208305 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:40.208425 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:40.209548 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0926 23:26:40.209755 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":244,"threads_started":1}
I20260926 23:26:40.211591 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:40.190317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:55071)
0926 23:26:40.191111 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:40.191153 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:40.193010 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:40.193821 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:40.193830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:40.194515 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:40.195165 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:40.195185 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.196494 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.196509 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:40.198105 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:40.198119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.199312 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.199317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:40.199531 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:40.200292 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:40.200320 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:40.202637 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:40.206358 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:40.206371 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:40.206394 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:40.207103 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:40.207558 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:40.207566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:40.207572 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:40.207849 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:40.208448 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:40.208461 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:40.209042 (+ 581us) client_negotiation.cc:769] Sending connection context
0926 23:26:40.209376 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 23:26:40.210011 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":303,"threads_started":1}
I20260926 23:26:40.213447 14461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37639
I20260926 23:26:40.213797 14461 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:40.214612 14461 heartbeater.cc:511] Master 127.0.97.62:37639 requested a full tablet report, sending...
I20260926 23:26:40.217216 14272 ts_manager.cc:194] Registered new tserver with Master: 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807)
I20260926 23:26:40.219566 14272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55071
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:26:40.251998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37639
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:40.598078 14467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:40.598367 14467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:40.598461 14467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:40.605901 14467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:40.606073 14467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:40.606195 14467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:26:40.616274 14467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:40.617673 14467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:40.619046 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:40.632580 14475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:40.632603 14473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:40.633091 14472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:40.634065 14467 server_base.cc:1034] running on GCE node
I20260926 23:26:40.634727 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:40.635761 14467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:40.636994 14467 hybrid_clock.cc:648] HybridClock initialized: now 1790465200636911 us; error 84 us; skew 500 ppm
I20260926 23:26:40.637346 14467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:40 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:40.640822 14467 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:26:40.642504 14467 webserver.cc:460] Webserver started at http://127.0.97.2:35031/ using document root <none> and password file <none>
I20260926 23:26:40.643262 14467 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:40.643417 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:40.643790 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:40.647238 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "4a61cd4c5afa46feb96642a8ae544a84"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "32d1b016fb067e4751c947096a6456b0"
server_key_iv: "3483b1ca8ef8c80024b1dc1747f2bc5a"
server_key_version: "encryptionkey@0"
I20260926 23:26:40.648118 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4a61cd4c5afa46feb96642a8ae544a84"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "32d1b016fb067e4751c947096a6456b0"
server_key_iv: "3483b1ca8ef8c80024b1dc1747f2bc5a"
server_key_version: "encryptionkey@0"
I20260926 23:26:40.653644 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 23:26:40.657414 14482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:40.658882 14467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:26:40.659067 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "4a61cd4c5afa46feb96642a8ae544a84"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "32d1b016fb067e4751c947096a6456b0"
server_key_iv: "3483b1ca8ef8c80024b1dc1747f2bc5a"
server_key_version: "encryptionkey@0"
I20260926 23:26:40.659297 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:40.676640 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:40.681094 14467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:40.681391 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:40.682353 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:40.683715 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:40.683818 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:40.683986 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:40.684059 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:40.718876 14467 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39729
I20260926 23:26:40.718971 14594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39729 every 8 connection(s)
I20260926 23:26:40.720224 14467 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:26:40.728734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 14467
I20260926 23:26:40.728965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:26:40.729568 388 external_mini_cluster.cc:1428] Setting key 18fb9a3cd12c546d7be36d23404e7c9a
Sep 26 23:26:40 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:40.744021 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:40.723249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46171 (local address 127.0.97.62:37639)
0926 23:26:40.723778 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:40.723784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:40.724930 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 23:26:40.727145 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:40.727151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:40.727255 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:40.727390 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:40.729386 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.730213 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.732087 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.732828 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.732997 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:40.736711 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:40.736721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:40.736728 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:40.736780 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:40.740214 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:40.741310 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:40.741315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:40.741317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:40.741458 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:40.741988 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:40.741991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:40.741993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:40.742272 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:40.742386 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:40.743433 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0926 23:26:40.743642 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
I20260926 23:26:40.745633 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:40.723853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:46171)
0926 23:26:40.724748 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:40.724791 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:40.726828 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:40.727649 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:40.727658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:40.728215 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:40.729171 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:40.729200 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.730399 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.730410 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:40.731919 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:40.731932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:40.733062 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:40.733067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:40.733299 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:40.734104 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:40.734129 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:40.736396 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:40.740394 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:40.740407 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:40.740429 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:40.741171 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:40.741579 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:40.741587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:40.741594 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:40.741870 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:40.742403 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:40.742415 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:40.742928 (+ 513us) client_negotiation.cc:769] Sending connection context
0926 23:26:40.743268 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 23:26:40.743925 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":337,"threads_started":1}
I20260926 23:26:40.747272 14595 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37639
I20260926 23:26:40.747615 14595 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:40.748545 14595 heartbeater.cc:511] Master 127.0.97.62:37639 requested a full tablet report, sending...
I20260926 23:26:40.750283 14272 ts_manager.cc:194] Registered new tserver with Master: 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729)
I20260926 23:26:40.751255 14272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46171
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:26:40.788934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37639
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:41.131202 14601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:41.131554 14601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:41.131731 14601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:41.139330 14601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:41.139513 14601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:41.139691 14601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:26:41.150041 14601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:41.151506 14601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:41.152859 14601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:41.167168 14606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:41.167102 14609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:41.167092 14607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:41.167614 14601 server_base.cc:1034] running on GCE node
I20260926 23:26:41.168509 14601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:41.169540 14601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:41.170799 14601 hybrid_clock.cc:648] HybridClock initialized: now 1790465201170737 us; error 46 us; skew 500 ppm
I20260926 23:26:41.171200 14601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:41 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:41.175093 14601 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:26:41.176672 14601 webserver.cc:460] Webserver started at http://127.0.97.3:35105/ using document root <none> and password file <none>
I20260926 23:26:41.177426 14601 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:41.177531 14601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:41.177783 14601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:26:41.180672 14601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "243d62011ce5442987cf11d2bf203785"
format_stamp: "Formatted at 2026-09-26 23:26:41 on dist-test-slave-q5g8"
server_key: "72efe5cbe5f58c62af7976c32110be7f"
server_key_iv: "f73d3af26614efda0b1c10cebb8222eb"
server_key_version: "encryptionkey@0"
I20260926 23:26:41.181478 14601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "243d62011ce5442987cf11d2bf203785"
format_stamp: "Formatted at 2026-09-26 23:26:41 on dist-test-slave-q5g8"
server_key: "72efe5cbe5f58c62af7976c32110be7f"
server_key_iv: "f73d3af26614efda0b1c10cebb8222eb"
server_key_version: "encryptionkey@0"
I20260926 23:26:41.186887 14601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:26:41.190495 14616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:41.191871 14601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:26:41.192065 14601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "243d62011ce5442987cf11d2bf203785"
format_stamp: "Formatted at 2026-09-26 23:26:41 on dist-test-slave-q5g8"
server_key: "72efe5cbe5f58c62af7976c32110be7f"
server_key_iv: "f73d3af26614efda0b1c10cebb8222eb"
server_key_version: "encryptionkey@0"
I20260926 23:26:41.192270 14601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:41.218135 14601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:41.222286 14601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:41.222618 14601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:41.223486 14461 heartbeater.cc:503] Master 127.0.97.62:37639 was elected leader, sending a full tablet report...
I20260926 23:26:41.223769 14601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:41.225448 14601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:41.225540 14601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:41.225699 14601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:41.225749 14601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:41.259831 14601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35275
I20260926 23:26:41.259912 14728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35275 every 8 connection(s)
I20260926 23:26:41.261238 14601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:26:41.263025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 14601
I20260926 23:26:41.263222 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:26:41.263769 388 external_mini_cluster.cc:1428] Setting key 58c5cfe1cfdfa64885535ce90b3a9455
Sep 26 23:26:41 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:41.282087 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.264155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34467 (local address 127.0.97.62:37639)
0926 23:26:41.264627 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.264633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.265893 (+ 1260us) server_negotiation.cc:407] Connection header received
0926 23:26:41.267737 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.267741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.267815 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.267914 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:41.269518 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.270286 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.271804 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.272279 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.272402 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.275996 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:41.276000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:41.276004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:41.276037 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:41.279032 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.279763 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.279768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.279772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.279900 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.280337 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.280341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.280345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.280619 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:41.280812 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.281501 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.281722 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":224,"threads_started":1}
I20260926 23:26:41.283326 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.264849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:34467)
0926 23:26:41.265745 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:41.265798 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:41.267492 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:41.268181 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:41.268189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:41.268764 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:41.269342 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.269358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.270421 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.270424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.271641 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.271656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.272489 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.272494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.272747 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.273455 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:41.273476 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:41.275791 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:41.279180 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:41.279190 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:41.279214 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:41.279637 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:41.280011 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:41.280016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:41.280020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:41.280205 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:41.280787 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:41.280794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:41.281137 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 23:26:41.281395 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 23:26:41.281923 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":396,"threads_started":1}
I20260926 23:26:41.284669 14729 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37639
I20260926 23:26:41.285094 14729 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:41.285754 14729 heartbeater.cc:511] Master 127.0.97.62:37639 requested a full tablet report, sending...
I20260926 23:26:41.287056 14272 ts_manager.cc:194] Registered new tserver with Master: 243d62011ce5442987cf11d2bf203785 (127.0.97.3:35275)
I20260926 23:26:41.287885 14272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34467
I20260926 23:26:41.291669 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:26:41.309290 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.296189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.0.97.62:37639)
0926 23:26:41.296383 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.296392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.296891 (+ 499us) server_negotiation.cc:407] Connection header received
0926 23:26:41.297099 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.297104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.297184 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.297289 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:41.298762 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.299498 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.301099 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.301509 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.301651 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.303969 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:41.303977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:41.303986 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:41.304026 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:41.306554 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.307258 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.307264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.307268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.307403 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.307792 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.307798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.307801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.308140 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:41.308406 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.308739 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.308972 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:26:41.313359 14272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:26:41.320276 14272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:26:41.344599 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.332916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35275 (local address 127.0.0.1:35644)
0926 23:26:41.334442 (+ 1526us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:41.334461 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:41.334580 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:41.335942 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:41.335951 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:41.335991 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:41.336495 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.336509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.338918 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.338925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.341644 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.341659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.343547 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.343554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.343762 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.343825 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 23:26:41.343937 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 23:26:41.344134 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"mutex_wait_us":25,"thread_start_us":1233,"threads_started":1}
I20260926 23:26:41.346354 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.332605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41807 (local address 127.0.0.1:44358)
0926 23:26:41.333778 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:41.333820 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:41.333982 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:41.336382 (+ 2400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:41.336389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:41.336422 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:41.336906 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.336919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.340164 (+ 3245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.340171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.342387 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.342401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.345483 (+ 3082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.345489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.345656 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.345687 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:26:41.345763 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 23:26:41.345913 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":1,"thread_start_us":804,"threads_started":1}
I20260926 23:26:41.346390 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.333575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.0.97.3:35275)
0926 23:26:41.335323 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.335333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.335363 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:26:41.335454 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.335462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.335713 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.336767 (+ 1054us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:41.337068 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.338749 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.341852 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.343328 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.345466 (+ 2138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.345738 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.345812 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.345981 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1573,"thread_start_us":1228,"threads_started":1}
I20260926 23:26:41.347460 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.332803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.0.97.1:41807)
0926 23:26:41.334660 (+ 1857us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.334671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.334702 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:26:41.334809 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.334819 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.335113 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.337106 (+ 1993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:41.337351 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.340000 (+ 2649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.342596 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.345256 (+ 2660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.346589 (+ 1333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.346918 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.346991 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.347169 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1685,"thread_start_us":1527,"threads_started":1}
I20260926 23:26:41.351105 14664 tablet_service.cc:1467] Processing CreateTablet for tablet 35a2248c13134c20ba566955ffda86ff (DEFAULT_TABLE table=test-table [id=a01050cad5424769bd2467df2afbd5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:41.352599 14664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a2248c13134c20ba566955ffda86ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:41.354570 14396 tablet_service.cc:1467] Processing CreateTablet for tablet 35a2248c13134c20ba566955ffda86ff (DEFAULT_TABLE table=test-table [id=a01050cad5424769bd2467df2afbd5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:41.355993 14396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a2248c13134c20ba566955ffda86ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:41.374130 14748 tablet_bootstrap.cc:492] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Bootstrap starting.
I20260926 23:26:41.378129 14749 tablet_bootstrap.cc:492] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Bootstrap starting.
I20260926 23:26:41.378244 14748 tablet_bootstrap.cc:654] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:41.379462 14748 log.cc:824] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:41.381925 14749 tablet_bootstrap.cc:654] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:41.381984 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.335229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39729 (local address 127.0.0.1:53490)
0926 23:26:41.345158 (+ 9929us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:41.345178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:41.345281 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:41.375770 (+ 30489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:41.375775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:41.375806 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:41.376011 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.376025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.377450 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.377457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.379384 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.379401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.381123 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.381133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.381317 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.381353 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:26:41.381458 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 23:26:41.381623 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9831,"thread_start_us":39533,"threads_started":1}
I20260926 23:26:41.382007 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.335312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53490 (local address 127.0.97.2:39729)
0926 23:26:41.375131 (+ 39819us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.375143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.375177 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:26:41.375279 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.375287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.375566 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.375779 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:41.376208 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.377302 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.379580 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.380925 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.381101 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.381301 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.381511 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.381690 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37696,"thread_start_us":37566,"threads_started":1}
I20260926 23:26:41.383037 14748 tablet_bootstrap.cc:492] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: No bootstrap required, opened a new log
I20260926 23:26:41.383185 14749 log.cc:824] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:41.383342 14748 ts_tablet_manager.cc:1397] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260926 23:26:41.386433 14749 tablet_bootstrap.cc:492] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: No bootstrap required, opened a new log
I20260926 23:26:41.386716 14749 ts_tablet_manager.cc:1397] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:26:41.386149 14530 tablet_service.cc:1467] Processing CreateTablet for tablet 35a2248c13134c20ba566955ffda86ff (DEFAULT_TABLE table=test-table [id=a01050cad5424769bd2467df2afbd5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:41.387646 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a2248c13134c20ba566955ffda86ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:41.393976 14748 raft_consensus.cc:348] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.394482 14748 raft_consensus.cc:374] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:41.394584 14748 raft_consensus.cc:729] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243d62011ce5442987cf11d2bf203785, State: Initialized, Role: FOLLOWER
I20260926 23:26:41.395264 14748 consensus_queue.cc:260] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.395999 14749 raft_consensus.cc:348] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.396737 14749 raft_consensus.cc:374] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:41.396945 14749 raft_consensus.cc:729] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26366434d62d4acda0642a34a218a82e, State: Initialized, Role: FOLLOWER
I20260926 23:26:41.397388 14748 ts_tablet_manager.cc:1428] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 23:26:41.397861 14729 heartbeater.cc:503] Master 127.0.97.62:37639 was elected leader, sending a full tablet report...
I20260926 23:26:41.398303 14749 consensus_queue.cc:260] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.399385 14752 raft_consensus.cc:482] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:26:41.399633 14752 raft_consensus.cc:504] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.404627 14749 ts_tablet_manager.cc:1428] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260926 23:26:41.407644 14755 tablet_bootstrap.cc:492] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Bootstrap starting.
I20260926 23:26:41.408427 14752 leader_election.cc:288] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807), 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729)
I20260926 23:26:41.411456 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.401826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41807 (local address 127.0.97.3:52563)
0926 23:26:41.402253 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:41.402280 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:41.402427 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:41.403666 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:41.403674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:41.403710 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:41.403958 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.403972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.405952 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.405960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.408627 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.408645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.410526 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.410533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.410756 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.410806 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 23:26:41.410929 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 23:26:41.411082 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260926 23:26:41.411506 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.402836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52563 (local address 127.0.97.1:41807)
0926 23:26:41.403116 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.403125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.403149 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:26:41.403235 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.403242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.403339 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.403448 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:41.404168 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.405735 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.408848 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.410174 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.410317 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.410436 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.411018 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.411193 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:26:41.412874 14416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a2248c13134c20ba566955ffda86ff" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26366434d62d4acda0642a34a218a82e" is_pre_election: true
I20260926 23:26:41.413625 14416 raft_consensus.cc:2454] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 243d62011ce5442987cf11d2bf203785 in term 0.
I20260926 23:26:41.414050 14755 tablet_bootstrap.cc:654] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:41.414623 14619 leader_election.cc:302] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243d62011ce5442987cf11d2bf203785, 26366434d62d4acda0642a34a218a82e; no voters:
I20260926 23:26:41.415151 14752 raft_consensus.cc:2781] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:26:41.415305 14752 raft_consensus.cc:482] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:26:41.415407 14752 raft_consensus.cc:3037] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:41.416371 14755 log.cc:824] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:41.419667 14752 raft_consensus.cc:504] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.420497 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.410329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54277 (local address 127.0.97.2:39729)
0926 23:26:41.410597 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.410603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.411817 (+ 1214us) server_negotiation.cc:407] Connection header received
0926 23:26:41.412095 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.412102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.412184 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.412249 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:41.412888 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.414147 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.416383 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.418345 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.418581 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.418758 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.419732 (+ 974us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.419981 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:26:41.421417 14416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a2248c13134c20ba566955ffda86ff" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26366434d62d4acda0642a34a218a82e"
I20260926 23:26:41.421614 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.410987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39729 (local address 127.0.97.3:54277)
0926 23:26:41.411718 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:41.411742 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:41.411934 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:41.412464 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:41.412470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:41.412500 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:41.412721 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.412733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.414290 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.414295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.416165 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:41.416180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.419021 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.419033 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:41.419396 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.419466 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 23:26:41.420752 (+ 1286us) client_negotiation.cc:240] Negotiation successful
0926 23:26:41.421030 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"threads_started":1}
I20260926 23:26:41.421711 14416 raft_consensus.cc:3037] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:41.425572 14752 leader_election.cc:288] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 election: Requested vote from peers 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807), 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729)
I20260926 23:26:41.425763 14416 raft_consensus.cc:2454] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 243d62011ce5442987cf11d2bf203785 in term 1.
I20260926 23:26:41.426314 14619 leader_election.cc:302] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243d62011ce5442987cf11d2bf203785, 26366434d62d4acda0642a34a218a82e; no voters:
I20260926 23:26:41.426673 14752 raft_consensus.cc:2781] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:26:41.427419 14752 raft_consensus.cc:686] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 1 LEADER]: Becoming Leader. State: Replica: 243d62011ce5442987cf11d2bf203785, State: Running, Role: LEADER
I20260926 23:26:41.428030 14752 consensus_queue.cc:237] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.429692 14755 tablet_bootstrap.cc:492] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: No bootstrap required, opened a new log
I20260926 23:26:41.430066 14755 ts_tablet_manager.cc:1397] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.002s
I20260926 23:26:41.427140 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a2248c13134c20ba566955ffda86ff" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a61cd4c5afa46feb96642a8ae544a84"
I20260926 23:26:41.426129 14550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a2248c13134c20ba566955ffda86ff" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a61cd4c5afa46feb96642a8ae544a84" is_pre_election: true
W20260926 23:26:41.433506 14618 leader_election.cc:341] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729): Illegal state: must be running to vote when last-logged opid is not known
W20260926 23:26:41.433993 14618 leader_election.cc:341] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729): Illegal state: must be running to vote when last-logged opid is not known
I20260926 23:26:41.434443 14271 catalog_manager.cc:5495] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 243d62011ce5442987cf11d2bf203785 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "243d62011ce5442987cf11d2bf203785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } health_report { overall_health: UNKNOWN } } }
W20260926 23:26:41.439229 14462 tablet.cc:2369] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:41.438916 14755 raft_consensus.cc:348] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.439512 14755 raft_consensus.cc:374] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:41.439599 14755 raft_consensus.cc:729] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a61cd4c5afa46feb96642a8ae544a84, State: Initialized, Role: FOLLOWER
I20260926 23:26:41.440255 14755 consensus_queue.cc:260] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } }
I20260926 23:26:41.441388 14595 heartbeater.cc:503] Master 127.0.97.62:37639 was elected leader, sending a full tablet report...
I20260926 23:26:41.442297 14755 ts_tablet_manager.cc:1428] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260926 23:26:41.471877 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.466471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.0.97.3:35275)
0926 23:26:41.466681 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.466687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.466708 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:26:41.466827 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.466831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.466909 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.467087 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:26:41.467475 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.468182 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.469682 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.470091 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.470229 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.470753 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:26:41.471266 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:26:41.471470 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.471521 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.471603 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 23:26:41.472438 14596 tablet.cc:2369] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:41.480255 14549 raft_consensus.cc:3037] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:41.480572 14416 raft_consensus.cc:1261] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Refusing update from remote peer 243d62011ce5442987cf11d2bf203785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:41.481588 14753 consensus_queue.cc:1035] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:41.483377 14549 raft_consensus.cc:1261] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [term 1 FOLLOWER]: Refusing update from remote peer 243d62011ce5442987cf11d2bf203785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:41.484237 14753 consensus_queue.cc:1035] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:41.492161 14762 mvcc.cc:205] Tried to move back new op lower bound from 7333745465252835328 to 7333745465054941184. Current Snapshot: MvccSnapshot[applied={T|T < 7333745465252835328}]
I20260926 23:26:41.523334 14271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:26:41.523581 14271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:26:41.527573 14271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Sending DeleteTablet for 3 replicas of tablet 35a2248c13134c20ba566955ffda86ff
I20260926 23:26:41.528823 14396 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a2248c13134c20ba566955ffda86ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44358
I20260926 23:26:41.528890 14664 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a2248c13134c20ba566955ffda86ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35644
I20260926 23:26:41.528951 14530 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a2248c13134c20ba566955ffda86ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53490
I20260926 23:26:41.529703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 14333
I20260926 23:26:41.540727 14772 tablet_replica.cc:330] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: stopping tablet replica
I20260926 23:26:41.541358 14772 raft_consensus.cc:2229] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:26:41.542194 14772 raft_consensus.cc:2258] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:41.545317 14772 ts_tablet_manager.cc:1905] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:41.545912 14770 tablet_replica.cc:330] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: stopping tablet replica
I20260926 23:26:41.546416 14770 raft_consensus.cc:2229] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:26:41.547124 14770 raft_consensus.cc:2258] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:41.549662 14770 ts_tablet_manager.cc:1905] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:26:41 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:26:41.556291 14772 ts_tablet_manager.cc:1918] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:41.556512 14772 log.cc:1196] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/35a2248c13134c20ba566955ffda86ff
I20260926 23:26:41.557199 14770 ts_tablet_manager.cc:1918] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:41.557358 14770 log.cc:1196] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/35a2248c13134c20ba566955ffda86ff
I20260926 23:26:41.557880 14770 ts_tablet_manager.cc:1939] T 35a2248c13134c20ba566955ffda86ff P 26366434d62d4acda0642a34a218a82e: Deleting consensus metadata
I20260926 23:26:41.560022 14258 catalog_manager.cc:4841] TS 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807): tablet 35a2248c13134c20ba566955ffda86ff (table test-table [id=a01050cad5424769bd2467df2afbd5a0]) successfully deleted
I20260926 23:26:41.561030 14771 tablet_replica.cc:330] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: stopping tablet replica
I20260926 23:26:41.561414 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.531272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.0.97.1:41807)
0926 23:26:41.531524 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.531533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.540355 (+ 8822us) server_negotiation.cc:407] Connection header received
0926 23:26:41.540834 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.540840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.540932 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.541012 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:41.543282 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.544461 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.546680 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.547386 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.547494 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.553147 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:41.553159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:41.553178 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:41.553225 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:41.557387 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.558568 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.558576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.558579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.558757 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.559265 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.559270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.559272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.559775 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:41.560049 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.560788 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.561030 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:26:41.561532 14771 raft_consensus.cc:2229] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:26:41.562052 14771 raft_consensus.cc:2258] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:41.562877 14456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:41.564230 14772 ts_tablet_manager.cc:1939] T 35a2248c13134c20ba566955ffda86ff P 243d62011ce5442987cf11d2bf203785: Deleting consensus metadata
I20260926 23:26:41.564675 14771 ts_tablet_manager.cc:1905] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:41.565923 14256 catalog_manager.cc:4841] TS 243d62011ce5442987cf11d2bf203785 (127.0.97.3:35275): tablet 35a2248c13134c20ba566955ffda86ff (table test-table [id=a01050cad5424769bd2467df2afbd5a0]) successfully deleted
I20260926 23:26:41.573637 14771 ts_tablet_manager.cc:1918] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:41.573836 14771 log.cc:1196] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/35a2248c13134c20ba566955ffda86ff
I20260926 23:26:41.574309 14771 ts_tablet_manager.cc:1939] T 35a2248c13134c20ba566955ffda86ff P 4a61cd4c5afa46feb96642a8ae544a84: Deleting consensus metadata
I20260926 23:26:41.575980 14257 catalog_manager.cc:4841] TS 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729): tablet 35a2248c13134c20ba566955ffda86ff (table test-table [id=a01050cad5424769bd2467df2afbd5a0]) successfully deleted
I20260926 23:26:41.680454 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 14333
I20260926 23:26:41.702435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 14467
Sep 26 23:26:41 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:26:41.719120 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.703273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.0.97.2:39729)
0926 23:26:41.703463 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.703469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.703489 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:26:41.703699 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.703705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.703787 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.703898 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:41.705277 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.706024 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.707340 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.707746 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.707878 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.712232 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:41.712251 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:41.712271 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:41.712315 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:41.715940 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.716565 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.716575 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.716579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.716772 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.717107 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.717113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.717117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.717583 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:41.717896 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.718494 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.718742 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:26:41.721131 14590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:41.815804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 14467
I20260926 23:26:41.839604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 14601
Sep 26 23:26:41 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:26:41.858152 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:41.840380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.0.97.3:35275)
0926 23:26:41.840563 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:41.840567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:41.840711 (+ 144us) server_negotiation.cc:407] Connection header received
0926 23:26:41.840905 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:41.840909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:41.840986 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:41.841105 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:41.842992 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.843872 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.845819 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:41.846336 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:41.846432 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:41.851774 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:41.851787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:41.851807 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:41.851867 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:41.855049 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.855776 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.855783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.855785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.855945 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:41.856379 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:41.856384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:41.856394 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:41.856863 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:41.857175 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:41.857491 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 23:26:41.857724 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:26:41.858867 14724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:41.984248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 14601
I20260926 23:26:42.004736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 14240
I20260926 23:26:42.005890 14302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:42.115506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 14240
I20260926 23:26:42.137032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37639
--webserver_interface=127.0.97.62
--webserver_port=35085
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:42.466917 14778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:42.467345 14778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:42.467517 14778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:42.474687 14778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:42.474848 14778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:42.474951 14778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:42.475057 14778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:42.475124 14778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:42.485100 14778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:42.486492 14778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:42.487984 14778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:42.511055 14784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:42.511116 14785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:42.511055 14787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:42.517634 14778 server_base.cc:1034] running on GCE node
I20260926 23:26:42.518893 14778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:42.521102 14778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:42.522387 14778 hybrid_clock.cc:648] HybridClock initialized: now 1790465202522318 us; error 50 us; skew 500 ppm
I20260926 23:26:42.522887 14778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:42 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:42.526521 14778 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:26:42.528270 14778 webserver.cc:460] Webserver started at http://127.0.97.62:35085/ using document root <none> and password file <none>
I20260926 23:26:42.529109 14778 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:42.529268 14778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:42.535146 14778 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 23:26:42.538082 14794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:42.539479 14778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:26:42.539722 14778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "935681ae62ee43a296022f52baa843ef"
format_stamp: "Formatted at 2026-09-26 23:26:39 on dist-test-slave-q5g8"
server_key: "f4e99196422e1ce87668e0fe99720654"
server_key_iv: "b26a84b830da5615c9e506bb2d660405"
server_key_version: "encryptionkey@0"
I20260926 23:26:42.540884 14778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:42.563843 14778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:42.567979 14778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:42.568272 14778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:42.587527 14778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37639
I20260926 23:26:42.587620 14845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37639 every 8 connection(s)
I20260926 23:26:42.589259 14778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:26:42.596400 14846 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:26:42.599692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 14778
I20260926 23:26:42.599910 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Bootstrap starting.
I20260926 23:26:42.600628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41807
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37639
--webserver_port=45061
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:42.620198 14846 log.cc:824] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:42.632229 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:26:42.632854 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Bootstrap complete.
I20260926 23:26:42.641925 14846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:42.642759 14846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935681ae62ee43a296022f52baa843ef, State: Initialized, Role: FOLLOWER
I20260926 23:26:42.643406 14846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:42.643669 14846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:26:42.643801 14846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:26:42.644001 14846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:26:42.647977 14846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:42.648475 14846 leader_election.cc:302] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 935681ae62ee43a296022f52baa843ef; no voters:
I20260926 23:26:42.649201 14846 leader_election.cc:288] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:26:42.649350 14850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:26:42.651736 14846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:42.656454 14850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 LEADER]: Becoming Leader. State: Replica: 935681ae62ee43a296022f52baa843ef, State: Running, Role: LEADER
I20260926 23:26:42.657176 14850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:42.661623 14851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "935681ae62ee43a296022f52baa843ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } } }
I20260926 23:26:42.662111 14851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:42.662734 14852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 935681ae62ee43a296022f52baa843ef. Latest consensus state: current_term: 2 leader_uuid: "935681ae62ee43a296022f52baa843ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } } }
I20260926 23:26:42.663059 14852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:42.666420 14864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:26:42.675926 14864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:26:42.677358 14864 catalog_manager.cc:1259] Loaded cluster ID: 9e3c92d4899641b88ac312eeac4eb34f
I20260926 23:26:42.677462 14864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:26:42.680534 14864 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 23:26:42.698697 14864 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 23:26:42.708731 14864 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:26:42.710354 14864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Loaded TSK: 0
I20260926 23:26:42.711834 14864 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:26:42.957576 14848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:42.957988 14848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:42.958177 14848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:42.965837 14848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:42.965999 14848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:42.966187 14848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:26:42.978204 14848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45061
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:42.979611 14848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:42.981092 14848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:42.995843 14874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:42.996595 14877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:42.996726 14875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:42.997043 14848 server_base.cc:1034] running on GCE node
I20260926 23:26:42.997769 14848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:42.998893 14848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:43.000128 14848 hybrid_clock.cc:648] HybridClock initialized: now 1790465203000060 us; error 58 us; skew 500 ppm
I20260926 23:26:43.000542 14848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:42 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:43.004446 14848 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:26:43.006093 14848 webserver.cc:460] Webserver started at http://127.0.97.1:45061/ using document root <none> and password file <none>
I20260926 23:26:43.006803 14848 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:43.006959 14848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:43.012761 14848 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:26:43.015713 14884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.017155 14848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:26:43.017351 14848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "26366434d62d4acda0642a34a218a82e"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "0d487b342a54c1c79fd78d12274d474c"
server_key_iv: "3f4f6c136eac6316ab28f03c856cf146"
server_key_version: "encryptionkey@0"
I20260926 23:26:43.018379 14848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:43.031121 14848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:43.035353 14848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:43.035633 14848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:43.036533 14848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:43.037964 14848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:43.038100 14848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.038234 14848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:43.038322 14848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.073515 14848 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41807
I20260926 23:26:43.073573 14996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41807 every 8 connection(s)
I20260926 23:26:43.075099 14848 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:26:43.084396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 14848
I20260926 23:26:43.085193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39729
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37639
--webserver_port=35031
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:26:43 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:43.099972 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:43.078225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:33605)
0926 23:26:43.078967 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:43.079001 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:43.080722 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:43.082107 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:43.082117 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:43.082716 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:43.083236 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:43.083252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.084967 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.084972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:43.086255 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:43.086267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.087266 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.087274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:43.087534 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:43.088218 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:43.088239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:43.090767 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:43.095371 (+ 4604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:43.095379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:43.095391 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:43.095879 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:43.096416 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:43.096420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:43.096422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:43.096625 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:43.097676 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:43.097683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:43.097997 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 23:26:43.098279 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 23:26:43.098680 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":312,"threads_started":1}
I20260926 23:26:43.100337 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:43.077912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33605 (local address 127.0.97.62:37639)
0926 23:26:43.078448 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:43.078460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:43.079131 (+ 671us) server_negotiation.cc:407] Connection header received
0926 23:26:43.081242 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:43.081255 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:43.081563 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:43.081919 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:43.083462 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.084807 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.086400 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.087030 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.087228 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:43.091032 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:43.091040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:43.091053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:43.091087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:43.095214 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.096032 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.096044 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.096056 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.096275 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.096825 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.096833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.096839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.097520 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:43.097728 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:43.098560 (+ 832us) server_negotiation.cc:299] Negotiation successful
0926 23:26:43.099040 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260926 23:26:43.103129 14997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37639
I20260926 23:26:43.103498 14997 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:43.104444 14997 heartbeater.cc:511] Master 127.0.97.62:37639 requested a full tablet report, sending...
I20260926 23:26:43.106815 14811 ts_manager.cc:194] Registered new tserver with Master: 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807)
I20260926 23:26:43.108309 14811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33605
W20260926 23:26:43.408946 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:43.409206 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:43.409381 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:43.416383 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:43.416486 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:43.416658 15001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:26:43.426442 15001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35031
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:43.427731 15001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:43.429083 15001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:43.442294 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:43.442445 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:43.442449 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:43.443602 15001 server_base.cc:1034] running on GCE node
I20260926 23:26:43.444172 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:43.445222 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:43.446452 15001 hybrid_clock.cc:648] HybridClock initialized: now 1790465203446374 us; error 80 us; skew 500 ppm
I20260926 23:26:43.446828 15001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:43 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:43.449977 15001 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:26:43.451463 15001 webserver.cc:460] Webserver started at http://127.0.97.2:35031/ using document root <none> and password file <none>
I20260926 23:26:43.452143 15001 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:43.452340 15001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:43.457828 15001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 23:26:43.460455 15016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.461851 15001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:26:43.462021 15001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "4a61cd4c5afa46feb96642a8ae544a84"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "32d1b016fb067e4751c947096a6456b0"
server_key_iv: "3483b1ca8ef8c80024b1dc1747f2bc5a"
server_key_version: "encryptionkey@0"
I20260926 23:26:43.462967 15001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:43.485575 15001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:43.489698 15001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:43.489984 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:43.490971 15001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:43.492370 15001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:43.492506 15001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.492635 15001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:43.492738 15001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.526584 15001 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39729
I20260926 23:26:43.526670 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39729 every 8 connection(s)
I20260926 23:26:43.528069 15001 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:26:43.530267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15001
I20260926 23:26:43.530929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35275
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37639
--webserver_port=35105
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:26:43 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:43.552157 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:43.531006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43357 (local address 127.0.97.62:37639)
0926 23:26:43.531262 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:43.531271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:43.532578 (+ 1307us) server_negotiation.cc:407] Connection header received
0926 23:26:43.535551 (+ 2973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:43.535556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:43.535626 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:43.535722 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:43.537837 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.538560 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.540081 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.540909 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.541052 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:43.544836 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:43.544844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:43.544852 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:43.544899 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:43.548538 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.549497 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.549506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.549513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.549689 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.550123 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.550131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.550137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.550562 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:43.550731 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:43.551419 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 23:26:43.551694 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 23:26:43.553159 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:43.531540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:43357)
0926 23:26:43.532389 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:43.532436 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:43.535249 (+ 2813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:43.535993 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:43.536006 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:43.536819 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:43.537648 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:43.537675 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.538728 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.538732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:43.539939 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:43.539951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.541171 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.541177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:43.541406 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:43.542140 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:43.542169 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:43.544519 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:43.548770 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:43.548777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:43.548792 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:43.549369 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:43.549810 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:43.549813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:43.549816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:43.550002 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:43.550693 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:43.550699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:43.551021 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 23:26:43.551256 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 23:26:43.551662 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":317,"threads_started":1}
I20260926 23:26:43.554736 15129 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37639
I20260926 23:26:43.555068 15129 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:43.555814 15129 heartbeater.cc:511] Master 127.0.97.62:37639 requested a full tablet report, sending...
I20260926 23:26:43.557363 14811 ts_manager.cc:194] Registered new tserver with Master: 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729)
I20260926 23:26:43.558328 14811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43357
W20260926 23:26:43.859575 15132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:43.859848 15132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:43.860008 15132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:43.867437 15132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:43.867542 15132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:43.867738 15132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:26:43.877372 15132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35105
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:43.878635 15132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:43.879927 15132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:43.892812 15137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:43.893070 15138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:43.893570 15140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:43.893800 15132 server_base.cc:1034] running on GCE node
I20260926 23:26:43.894429 15132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:43.895493 15132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:43.896780 15132 hybrid_clock.cc:648] HybridClock initialized: now 1790465203896723 us; error 72 us; skew 500 ppm
I20260926 23:26:43.897141 15132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:43 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:43.900302 15132 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:26:43.901721 15132 webserver.cc:460] Webserver started at http://127.0.97.3:35105/ using document root <none> and password file <none>
I20260926 23:26:43.902356 15132 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:43.902465 15132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:43.907618 15132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 23:26:43.910187 15147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.911523 15132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 23:26:43.911670 15132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "243d62011ce5442987cf11d2bf203785"
format_stamp: "Formatted at 2026-09-26 23:26:41 on dist-test-slave-q5g8"
server_key: "72efe5cbe5f58c62af7976c32110be7f"
server_key_iv: "f73d3af26614efda0b1c10cebb8222eb"
server_key_version: "encryptionkey@0"
I20260926 23:26:43.912627 15132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:43.932909 15132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:43.937274 15132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:43.937505 15132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:43.938400 15132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:43.939656 15132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:43.939739 15132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.939816 15132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:43.939870 15132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:43.972530 15132 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35275
I20260926 23:26:43.972659 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35275 every 8 connection(s)
I20260926 23:26:43.974025 15132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:26:43.976612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15132
Sep 26 23:26:43 dist-test-slave-q5g8 krb5kdc[14227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:26:43.994307 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:43.977992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.0.97.62:37639)
0926 23:26:43.979998 (+ 2006us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:43.980002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:43.980017 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:26:43.980067 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:43.980071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:43.980128 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:43.980222 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:43.982205 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.982921 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.984775 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.985167 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.985266 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:43.988418 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:43.988422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:43.988425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:43.988452 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:43.991399 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.992048 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.992059 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.992062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.992203 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.992637 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.992642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.992646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.993072 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:43.993335 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:43.993582 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 23:26:43.993824 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1902,"thread_start_us":1811,"threads_started":1}
I20260926 23:26:43.997442 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:43.976957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50105 (local address 127.0.97.62:37639)
0926 23:26:43.977148 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:43.977154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:43.979262 (+ 2108us) server_negotiation.cc:407] Connection header received
0926 23:26:43.981656 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:43.981660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:43.981710 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:43.981782 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:43.983442 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.984152 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.986397 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.986815 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.986900 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:43.990837 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:43.990841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:43.990844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:43.990886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:43.994118 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.995010 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.995017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.995021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.995156 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:43.995698 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:43.995703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:43.995707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:43.996016 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:43.996127 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:43.997061 (+ 934us) server_negotiation.cc:299] Negotiation successful
0926 23:26:43.997224 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:26:43.998512 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:43.977508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:50105)
0926 23:26:43.979096 (+ 1588us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:43.979155 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:43.981354 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:43.982038 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:43.982049 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:43.982597 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:26:43.983267 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:43.983293 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.984381 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.984388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:43.986256 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:43.986272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:43.987062 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:43.987071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:43.987385 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:43.988047 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:26:43.988084 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:26:43.990610 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:26:43.994268 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:43.994274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:43.994286 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:43.994867 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:43.995269 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:26:43.995273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:26:43.995275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:26:43.995538 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:26:43.996152 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:26:43.996159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:26:43.996599 (+ 440us) client_negotiation.cc:769] Sending connection context
0926 23:26:43.996943 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 23:26:43.997402 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":1047,"threads_started":1}
I20260926 23:26:43.999871 15260 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37639
I20260926 23:26:44.000141 15260 heartbeater.cc:464] Registering TS with master...
I20260926 23:26:44.000895 15260 heartbeater.cc:511] Master 127.0.97.62:37639 requested a full tablet report, sending...
I20260926 23:26:44.002262 14811 ts_manager.cc:194] Registered new tserver with Master: 243d62011ce5442987cf11d2bf203785 (127.0.97.3:35275)
I20260926 23:26:44.003053 14811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50105
I20260926 23:26:44.009357 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:26:44.016386 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.010326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.0.97.62:37639)
0926 23:26:44.010520 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.010527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.011072 (+ 545us) server_negotiation.cc:407] Connection header received
0926 23:26:44.011304 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.011309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.011512 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.011672 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:26:44.012081 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.012887 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.014316 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.014742 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.014866 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.015354 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:26:44.015828 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:26:44.015979 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.016065 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.016150 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:26:44.017510 14811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:26:44.024730 14811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:26:44.067128 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.036628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39729 (local address 127.0.0.1:53518)
0926 23:26:44.057387 (+ 20759us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:44.057415 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:44.057578 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:44.060339 (+ 2761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:44.060346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:44.060377 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:44.060746 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.060764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.062193 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.062198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.064379 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.064403 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.066077 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.066083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.066283 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.066423 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 23:26:44.066557 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 23:26:44.066743 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20532,"thread_start_us":23024,"threads_started":1}
I20260926 23:26:44.067171 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.037459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.0.97.2:39729)
0926 23:26:44.059525 (+ 22066us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.059539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.059585 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:26:44.059734 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.059745 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.060073 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.060299 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:44.060908 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.062039 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.064561 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.065886 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.066018 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.066207 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.066650 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.066833 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21849,"thread_start_us":21720,"threads_started":1}
I20260926 23:26:44.068349 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.036428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41807 (local address 127.0.0.1:44376)
0926 23:26:44.037366 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:44.037422 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:44.037628 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:44.060991 (+ 23363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:44.060997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:44.061022 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:44.061452 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.061467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.063544 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.063549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.065647 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.065656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.067494 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.067503 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.067744 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.067788 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 23:26:44.067851 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 23:26:44.067979 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":1166,"threads_started":1}
I20260926 23:26:44.068338 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.039445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.0.97.3:35275)
0926 23:26:44.051954 (+ 12509us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.051962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.060115 (+ 8153us) server_negotiation.cc:407] Connection header received
0926 23:26:44.060448 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.060457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.060757 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.060921 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:44.061865 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.062977 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.065559 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.066896 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.067022 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.067253 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.067835 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.067993 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12330,"thread_start_us":12169,"threads_started":1}
I20260926 23:26:44.068574 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.039052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35275 (local address 127.0.0.1:35674)
0926 23:26:44.059975 (+ 20923us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:44.059999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:44.060228 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:44.061248 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:44.061255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:44.061285 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:44.061686 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.061697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.063114 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.063120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.065400 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.065414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.067431 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.067437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.067642 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.067686 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 23:26:44.067770 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 23:26:44.067908 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20770,"mutex_wait_us":5,"thread_start_us":17932,"threads_started":1}
I20260926 23:26:44.069315 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.036600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.0.97.1:41807)
0926 23:26:44.060354 (+ 23754us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.060365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.060394 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:26:44.060475 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.060482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.060748 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.061684 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:44.061912 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.063397 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.065790 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.067220 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.068614 (+ 1394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.068849 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.068914 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.069066 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23542,"thread_start_us":23399,"threads_started":1}
I20260926 23:26:44.071810 15064 tablet_service.cc:1467] Processing CreateTablet for tablet c1660bab834448ca825e1c7d1823ee22 (DEFAULT_TABLE table=test-table [id=6e02f217ad894fdeab1d934561cc560d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:44.072206 15195 tablet_service.cc:1467] Processing CreateTablet for tablet c1660bab834448ca825e1c7d1823ee22 (DEFAULT_TABLE table=test-table [id=6e02f217ad894fdeab1d934561cc560d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:44.072350 14932 tablet_service.cc:1467] Processing CreateTablet for tablet c1660bab834448ca825e1c7d1823ee22 (DEFAULT_TABLE table=test-table [id=6e02f217ad894fdeab1d934561cc560d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:26:44.073417 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1660bab834448ca825e1c7d1823ee22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:44.073685 14932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1660bab834448ca825e1c7d1823ee22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:44.073836 15195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1660bab834448ca825e1c7d1823ee22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:26:44.088569 15276 tablet_bootstrap.cc:492] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Bootstrap starting.
I20260926 23:26:44.090567 15277 tablet_bootstrap.cc:492] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Bootstrap starting.
I20260926 23:26:44.092234 15278 tablet_bootstrap.cc:492] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Bootstrap starting.
I20260926 23:26:44.092916 15276 tablet_bootstrap.cc:654] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:44.094269 15276 log.cc:824] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:44.094977 15277 tablet_bootstrap.cc:654] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:44.095264 15278 tablet_bootstrap.cc:654] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Neither blocks nor log segments found. Creating new log.
I20260926 23:26:44.096131 15277 log.cc:824] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:44.096318 15278 log.cc:824] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:44.097684 15276 tablet_bootstrap.cc:492] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: No bootstrap required, opened a new log
I20260926 23:26:44.097968 15276 ts_tablet_manager.cc:1397] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 23:26:44.099113 15277 tablet_bootstrap.cc:492] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: No bootstrap required, opened a new log
I20260926 23:26:44.099138 15278 tablet_bootstrap.cc:492] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: No bootstrap required, opened a new log
I20260926 23:26:44.099386 15278 ts_tablet_manager.cc:1397] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 23:26:44.099383 15277 ts_tablet_manager.cc:1397] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:26:44.105623 15276 raft_consensus.cc:348] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.105997 15276 raft_consensus.cc:374] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:44.106101 15276 raft_consensus.cc:729] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26366434d62d4acda0642a34a218a82e, State: Initialized, Role: FOLLOWER
I20260926 23:26:44.106715 15276 consensus_queue.cc:260] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.107846 14997 heartbeater.cc:503] Master 127.0.97.62:37639 was elected leader, sending a full tablet report...
I20260926 23:26:44.108518 15276 ts_tablet_manager.cc:1428] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260926 23:26:44.110208 15277 raft_consensus.cc:348] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.110215 15278 raft_consensus.cc:348] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.110656 15277 raft_consensus.cc:374] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:44.110651 15278 raft_consensus.cc:374] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:26:44.110769 15278 raft_consensus.cc:729] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243d62011ce5442987cf11d2bf203785, State: Initialized, Role: FOLLOWER
I20260926 23:26:44.110769 15277 raft_consensus.cc:729] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a61cd4c5afa46feb96642a8ae544a84, State: Initialized, Role: FOLLOWER
I20260926 23:26:44.111361 15278 consensus_queue.cc:260] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.111361 15277 consensus_queue.cc:260] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.113526 15278 ts_tablet_manager.cc:1428] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 23:26:44.114012 15129 heartbeater.cc:503] Master 127.0.97.62:37639 was elected leader, sending a full tablet report...
I20260926 23:26:44.114061 15277 ts_tablet_manager.cc:1428] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 23:26:44.114073 15260 heartbeater.cc:503] Master 127.0.97.62:37639 was elected leader, sending a full tablet report...
I20260926 23:26:44.207543 15284 raft_consensus.cc:482] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:26:44.207880 15284 raft_consensus.cc:504] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.209654 15284 leader_election.cc:288] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807), 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729)
I20260926 23:26:44.217018 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.210076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60421 (local address 127.0.97.1:41807)
0926 23:26:44.210468 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.210477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.210507 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:26:44.210743 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.210748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.210832 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.210931 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:44.211399 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.212413 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.214891 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.216047 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.216180 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.216277 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.216583 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.216765 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260926 23:26:44.217064 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.210021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41807 (local address 127.0.97.3:60421)
0926 23:26:44.210404 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:44.210424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:44.210562 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:44.210991 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:44.211000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:44.211034 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:44.211261 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.211270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.212533 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.212536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.214738 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.214753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.216181 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.216185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.216393 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.216418 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:26:44.216573 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 23:26:44.216743 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":57}
I20260926 23:26:44.217331 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.210189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56635 (local address 127.0.97.2:39729)
0926 23:26:44.210406 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.210414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.211076 (+ 662us) server_negotiation.cc:407] Connection header received
0926 23:26:44.211218 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.211229 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.211283 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.211388 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:26:44.211848 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.212793 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.215101 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.216362 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.216471 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.216586 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.216941 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.217074 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 23:26:44.217506 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.210138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39729 (local address 127.0.97.3:56635)
0926 23:26:44.210965 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:44.210980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:26:44.211101 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:26:44.211417 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:26:44.211420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:26:44.211438 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:26:44.211695 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.211701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.212949 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.212954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.214970 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:26:44.214979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.216578 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.216584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:26:44.216783 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.216811 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 23:26:44.216900 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 23:26:44.217007 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":477,"threads_started":1}
I20260926 23:26:44.218155 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1660bab834448ca825e1c7d1823ee22" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26366434d62d4acda0642a34a218a82e" is_pre_election: true
I20260926 23:26:44.218510 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1660bab834448ca825e1c7d1823ee22" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a61cd4c5afa46feb96642a8ae544a84" is_pre_election: true
I20260926 23:26:44.218731 14952 raft_consensus.cc:2454] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 243d62011ce5442987cf11d2bf203785 in term 0.
I20260926 23:26:44.218986 15084 raft_consensus.cc:2454] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 243d62011ce5442987cf11d2bf203785 in term 0.
I20260926 23:26:44.219533 15150 leader_election.cc:302] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243d62011ce5442987cf11d2bf203785, 26366434d62d4acda0642a34a218a82e; no voters:
I20260926 23:26:44.219974 15284 raft_consensus.cc:2781] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:26:44.220117 15284 raft_consensus.cc:482] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:26:44.220222 15284 raft_consensus.cc:3037] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:44.222525 15284 raft_consensus.cc:504] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.223114 15284 leader_election.cc:288] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 election: Requested vote from peers 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807), 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729)
I20260926 23:26:44.223482 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1660bab834448ca825e1c7d1823ee22" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26366434d62d4acda0642a34a218a82e"
I20260926 23:26:44.223717 14952 raft_consensus.cc:3037] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:26:44.223722 15084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1660bab834448ca825e1c7d1823ee22" candidate_uuid: "243d62011ce5442987cf11d2bf203785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a61cd4c5afa46feb96642a8ae544a84"
I20260926 23:26:44.223907 15084 raft_consensus.cc:3037] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 0 FOLLOWER]: Advancing to term 1
W20260926 23:26:44.225577 15261 tablet.cc:2369] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:44.226042 14952 raft_consensus.cc:2454] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 243d62011ce5442987cf11d2bf203785 in term 1.
I20260926 23:26:44.226539 15150 leader_election.cc:302] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243d62011ce5442987cf11d2bf203785, 26366434d62d4acda0642a34a218a82e; no voters:
I20260926 23:26:44.226594 15084 raft_consensus.cc:2454] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 243d62011ce5442987cf11d2bf203785 in term 1.
I20260926 23:26:44.226819 15284 raft_consensus.cc:2781] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:26:44.227252 15284 raft_consensus.cc:686] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 1 LEADER]: Becoming Leader. State: Replica: 243d62011ce5442987cf11d2bf203785, State: Running, Role: LEADER
I20260926 23:26:44.227694 15284 consensus_queue.cc:237] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } }
I20260926 23:26:44.231842 14810 catalog_manager.cc:5495] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 reported cstate change: term changed from 0 to 1, leader changed from <none> to 243d62011ce5442987cf11d2bf203785 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "243d62011ce5442987cf11d2bf203785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "243d62011ce5442987cf11d2bf203785" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35275 } health_report { overall_health: HEALTHY } } }
W20260926 23:26:44.280207 15130 tablet.cc:2369] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:26:44.291479 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.285915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.0.97.3:35275)
0926 23:26:44.286116 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.286124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.286163 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:26:44.286416 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.286420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.286505 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.286649 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:26:44.287020 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.287802 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.289365 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.289758 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.289883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.290392 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:26:44.290916 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:26:44.291073 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.291132 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.291211 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:26:44.299669 15084 raft_consensus.cc:1261] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 1 FOLLOWER]: Refusing update from remote peer 243d62011ce5442987cf11d2bf203785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:44.299731 14952 raft_consensus.cc:1261] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Refusing update from remote peer 243d62011ce5442987cf11d2bf203785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:26:44.300652 15286 consensus_queue.cc:1035] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a61cd4c5afa46feb96642a8ae544a84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:44.301038 15284 consensus_queue.cc:1035] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26366434d62d4acda0642a34a218a82e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:26:44.312862 15288 mvcc.cc:205] Tried to move back new op lower bound from 7333745476802076672 to 7333745476520087552. Current Snapshot: MvccSnapshot[applied={T|T < 7333745476802076672}]
I20260926 23:26:44.331959 14810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:26:44.332285 14810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:26:44.332535 15289 mvcc.cc:205] Tried to move back new op lower bound from 7333745476802076672 to 7333745476520087552. Current Snapshot: MvccSnapshot[applied={T|T < 7333745476802076672}]
I20260926 23:26:44.337608 14810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Sending DeleteTablet for 3 replicas of tablet c1660bab834448ca825e1c7d1823ee22
I20260926 23:26:44.338845 14932 tablet_service.cc:1514] Processing DeleteTablet for tablet c1660bab834448ca825e1c7d1823ee22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44376
I20260926 23:26:44.338889 15064 tablet_service.cc:1514] Processing DeleteTablet for tablet c1660bab834448ca825e1c7d1823ee22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53518
I20260926 23:26:44.339677 15298 tablet_replica.cc:330] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: stopping tablet replica
I20260926 23:26:44.339740 15195 tablet_service.cc:1514] Processing DeleteTablet for tablet c1660bab834448ca825e1c7d1823ee22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35674
I20260926 23:26:44.340179 15298 raft_consensus.cc:2229] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:26:44.340310 15063 tablet_service.cc:1514] Processing DeleteTablet for tablet c1660bab834448ca825e1c7d1823ee22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53518
I20260926 23:26:44.340646 15194 tablet_service.cc:1514] Processing DeleteTablet for tablet c1660bab834448ca825e1c7d1823ee22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:26:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35674
I20260926 23:26:44.340929 15298 raft_consensus.cc:2258] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:44.341691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 14848
I20260926 23:26:44.343076 15298 ts_tablet_manager.cc:1905] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:44.343477 15299 tablet_replica.cc:330] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: stopping tablet replica
I20260926 23:26:44.343914 15299 raft_consensus.cc:2229] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:26:44.344434 15299 raft_consensus.cc:2258] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:44.349154 15300 tablet_replica.cc:330] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: stopping tablet replica
I20260926 23:26:44.349328 15298 ts_tablet_manager.cc:1918] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:44.349547 15300 raft_consensus.cc:2229] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:26:44.349763 15298 log.cc:1196] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/c1660bab834448ca825e1c7d1823ee22
I20260926 23:26:44.350142 15300 raft_consensus.cc:2258] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:26:44.350425 15298 ts_tablet_manager.cc:1939] T c1660bab834448ca825e1c7d1823ee22 P 26366434d62d4acda0642a34a218a82e: Deleting consensus metadata
I20260926 23:26:44.351398 15299 ts_tablet_manager.cc:1905] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:44.352167 14797 catalog_manager.cc:4841] TS 26366434d62d4acda0642a34a218a82e (127.0.97.1:41807): tablet c1660bab834448ca825e1c7d1823ee22 (table test-table [id=6e02f217ad894fdeab1d934561cc560d]) successfully deleted
I20260926 23:26:44.352612 15300 ts_tablet_manager.cc:1905] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:26:44.358327 15300 ts_tablet_manager.cc:1918] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:44.358532 15300 log.cc:1196] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/c1660bab834448ca825e1c7d1823ee22
I20260926 23:26:44.359100 15300 ts_tablet_manager.cc:1939] T c1660bab834448ca825e1c7d1823ee22 P 243d62011ce5442987cf11d2bf203785: Deleting consensus metadata
I20260926 23:26:44.360899 14795 catalog_manager.cc:4841] TS 243d62011ce5442987cf11d2bf203785 (127.0.97.3:35275): tablet c1660bab834448ca825e1c7d1823ee22 (table test-table [id=6e02f217ad894fdeab1d934561cc560d]) successfully deleted
W20260926 23:26:44.361382 14795 catalog_manager.cc:4808] TS 243d62011ce5442987cf11d2bf203785 (127.0.97.3:35275): delete failed for tablet c1660bab834448ca825e1c7d1823ee22 because the tablet was not found. No further retry: Not found: Tablet not found: c1660bab834448ca825e1c7d1823ee22
I20260926 23:26:44.363081 15299 ts_tablet_manager.cc:1918] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:26:44.363255 15299 log.cc:1196] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/c1660bab834448ca825e1c7d1823ee22
I20260926 23:26:44.363816 15299 ts_tablet_manager.cc:1939] T c1660bab834448ca825e1c7d1823ee22 P 4a61cd4c5afa46feb96642a8ae544a84: Deleting consensus metadata
I20260926 23:26:44.364745 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.342495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.97.1:41807)
0926 23:26:44.342655 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.342664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.342761 (+ 97us) server_negotiation.cc:407] Connection header received
0926 23:26:44.342959 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.342964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.343044 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.343161 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:44.345420 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.346563 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.348995 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.349701 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.349857 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.353485 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:44.353498 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:44.353515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:44.353555 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:44.357407 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:44.362056 (+ 4649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:44.362066 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:44.362069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:44.362225 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:44.362676 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:44.362682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:44.362686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:44.363178 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:44.363494 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.363933 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.364182 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:26:44.365427 14992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:44.365561 14796 catalog_manager.cc:4841] TS 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729): tablet c1660bab834448ca825e1c7d1823ee22 (table test-table [id=6e02f217ad894fdeab1d934561cc560d]) successfully deleted
W20260926 23:26:44.365993 14796 catalog_manager.cc:4808] TS 4a61cd4c5afa46feb96642a8ae544a84 (127.0.97.2:39729): delete failed for tablet c1660bab834448ca825e1c7d1823ee22 because the tablet was not found. No further retry: Not found: Tablet not found: c1660bab834448ca825e1c7d1823ee22
I20260926 23:26:44.455422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 14848
I20260926 23:26:44.473219 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 15001
I20260926 23:26:44.490710 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.474130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.0.97.2:39729)
0926 23:26:44.474316 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.474321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.474341 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:26:44.474492 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.474497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.474578 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.474682 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:44.476019 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.476941 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.478350 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.478765 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.478896 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.485179 (+ 6283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:44.485193 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:44.485209 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:44.485243 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:44.487980 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:44.488669 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:44.488676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:44.488715 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:44.488856 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:44.489210 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:44.489217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:44.489220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:44.489571 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:44.489889 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.490170 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.490390 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 23:26:44.491468 15124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:44.581300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 15001
I20260926 23:26:44.604895 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 15132
I20260926 23:26:44.623152 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:44.605783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.0.97.3:35275)
0926 23:26:44.605995 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:26:44.606001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:26:44.606024 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:26:44.606224 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:26:44.606231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:26:44.606325 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:26:44.606454 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:26:44.608084 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.609468 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.611476 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:26:44.612188 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:26:44.612332 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:26:44.615630 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:26:44.615650 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:26:44.615671 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:26:44.615725 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:26:44.620131 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:44.620908 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:44.620918 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:44.620922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:44.621102 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:26:44.621460 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:26:44.621468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:26:44.621473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:26:44.621953 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:26:44.622285 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 23:26:44.622463 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 23:26:44.622697 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:26:44.623998 15255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:44.744237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 15132
I20260926 23:26:44.761264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 14778
I20260926 23:26:44.762422 14841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:26:44.874583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 14778
I20260926 23:26:44.896600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37639
--webserver_interface=127.0.97.62
--webserver_port=35085
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:26:45.229378 15305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:45.229720 15305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:45.229820 15305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:45.237160 15305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:26:45.237267 15305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:45.237317 15305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:45.237372 15305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:26:45.237423 15305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:26:45.247280 15305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:45.248536 15305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:45.249967 15305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:45.261880 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:45.261915 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:45.261941 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:45.262745 15305 server_base.cc:1034] running on GCE node
I20260926 23:26:45.263450 15305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:45.264871 15305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:45.266064 15305 hybrid_clock.cc:648] HybridClock initialized: now 1790465205266014 us; error 52 us; skew 500 ppm
I20260926 23:26:45.266347 15305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:45 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:45.269512 15305 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:26:45.270955 15305 webserver.cc:460] Webserver started at http://127.0.97.62:35085/ using document root <none> and password file <none>
I20260926 23:26:45.271574 15305 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:45.271677 15305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:45.277144 15305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:26:45.279960 15320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:45.281392 15305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:26:45.281572 15305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "935681ae62ee43a296022f52baa843ef"
format_stamp: "Formatted at 2026-09-26 23:26:39 on dist-test-slave-q5g8"
server_key: "f4e99196422e1ce87668e0fe99720654"
server_key_iv: "b26a84b830da5615c9e506bb2d660405"
server_key_version: "encryptionkey@0"
I20260926 23:26:45.282531 15305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:45.298264 15305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:45.302192 15305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:45.302469 15305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:45.320921 15305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37639
I20260926 23:26:45.320945 15371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37639 every 8 connection(s)
I20260926 23:26:45.322400 15305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:26:45.328368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15305
I20260926 23:26:45.329372 15372 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:26:45.329422 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41807
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37639
--webserver_port=45061
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:45.334523 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Bootstrap starting.
I20260926 23:26:45.355571 15372 log.cc:824] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Log is configured to *not* fsync() on all Append() calls
I20260926 23:26:45.371234 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:26:45.371778 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef: Bootstrap complete.
I20260926 23:26:45.380795 15372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:45.381654 15372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935681ae62ee43a296022f52baa843ef, State: Initialized, Role: FOLLOWER
I20260926 23:26:45.382289 15372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:45.382519 15372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:26:45.382627 15372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:26:45.382820 15372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:26:45.386619 15372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:45.387127 15372 leader_election.cc:302] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 935681ae62ee43a296022f52baa843ef; no voters:
I20260926 23:26:45.387782 15372 leader_election.cc:288] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 23:26:45.387907 15376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 3 FOLLOWER]: Leader election won for term 3
I20260926 23:26:45.390158 15372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:26:45.394726 15376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [term 3 LEADER]: Becoming Leader. State: Replica: 935681ae62ee43a296022f52baa843ef, State: Running, Role: LEADER
I20260926 23:26:45.395339 15376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } }
I20260926 23:26:45.400959 15378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 935681ae62ee43a296022f52baa843ef. Latest consensus state: current_term: 3 leader_uuid: "935681ae62ee43a296022f52baa843ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } } }
I20260926 23:26:45.401440 15378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:45.402047 15377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "935681ae62ee43a296022f52baa843ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935681ae62ee43a296022f52baa843ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37639 } } }
I20260926 23:26:45.402369 15377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935681ae62ee43a296022f52baa843ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:26:45.404532 15386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:26:45.416438 15386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:26:45.418277 15386 catalog_manager.cc:1259] Loaded cluster ID: 9e3c92d4899641b88ac312eeac4eb34f
I20260926 23:26:45.418371 15386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 23:26:45.420223 15386 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff02afc79ad google::LogMessage::Fail() at ??:0
@ 0x7ff02afca88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff02afc74de google::LogMessage::Flush() at ??:0
@ 0x7ff02afc8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff0409fa512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff0409f7976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff040a7f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff040a7f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff02dc13345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff02dc1691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff02dc16799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff02dbf1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff030ecb6db start_thread at ??:0
@ 0x7ff029e6161f clone at ??:0
W20260926 23:26:45.726063 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:45.726361 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:45.726549 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:45.734078 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:45.734195 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:45.734382 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:26:45.744418 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45061
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:45.745780 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:45.747120 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:45.760859 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:45.760762 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:45.760888 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:45.762042 15374 server_base.cc:1034] running on GCE node
I20260926 23:26:45.762698 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:45.763638 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:45.764873 15374 hybrid_clock.cc:648] HybridClock initialized: now 1790465205764831 us; error 53 us; skew 500 ppm
I20260926 23:26:45.765226 15374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:45 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:45.768563 15374 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:26:45.770113 15374 webserver.cc:460] Webserver started at http://127.0.97.1:45061/ using document root <none> and password file <none>
I20260926 23:26:45.770820 15374 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:45.770979 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:45.776417 15374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:26:45.779243 15424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:45.780622 15374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:26:45.780830 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "26366434d62d4acda0642a34a218a82e"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "0d487b342a54c1c79fd78d12274d474c"
server_key_iv: "3f4f6c136eac6316ab28f03c856cf146"
server_key_version: "encryptionkey@0"
I20260926 23:26:45.781847 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:45.810096 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:45.814191 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:45.814522 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:45.815550 15374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:45.816960 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:45.817087 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:45.817229 15374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:45.817327 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:45.851228 15374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41807
I20260926 23:26:45.851294 15536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41807 every 8 connection(s)
I20260926 23:26:45.852944 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:26:45.855985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15374
I20260926 23:26:45.856588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39729
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37639
--webserver_port=35031
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:45.858158 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:45.855940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:40115)
0926 23:26:45.856860 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:45.857128 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":380,"threads_started":1}
W20260926 23:26:45.858781 15537 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37639 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37639: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
I20260926 23:26:45.859689 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:45.859355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:32977)
0926 23:26:45.859504 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:45.859587 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 23:26:45.860597 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:45.860302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:50685)
0926 23:26:45.860433 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:45.860500 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 23:26:45.860895 15537 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:26:46.184453 15540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:46.184823 15540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:46.184929 15540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:46.192427 15540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:46.192548 15540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:46.192662 15540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:26:46.202800 15540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35031
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:46.204057 15540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:46.205433 15540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:46.218854 15545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:46.218741 15546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:46.218902 15548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:46.219599 15540 server_base.cc:1034] running on GCE node
I20260926 23:26:46.220150 15540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:46.221163 15540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:46.222376 15540 hybrid_clock.cc:648] HybridClock initialized: now 1790465206222309 us; error 54 us; skew 500 ppm
I20260926 23:26:46.222679 15540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:46 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:46.226037 15540 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:26:46.227573 15540 webserver.cc:460] Webserver started at http://127.0.97.2:35031/ using document root <none> and password file <none>
I20260926 23:26:46.228274 15540 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:46.228420 15540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:46.233717 15540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:26:46.236416 15555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:46.237762 15540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:26:46.237957 15540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "4a61cd4c5afa46feb96642a8ae544a84"
format_stamp: "Formatted at 2026-09-26 23:26:40 on dist-test-slave-q5g8"
server_key: "32d1b016fb067e4751c947096a6456b0"
server_key_iv: "3483b1ca8ef8c80024b1dc1747f2bc5a"
server_key_version: "encryptionkey@0"
I20260926 23:26:46.238926 15540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:46.271791 15540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:46.276520 15540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:46.276853 15540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:46.277949 15540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:46.279433 15540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:46.279556 15540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:46.279693 15540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:46.279779 15540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:46.314927 15540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39729
I20260926 23:26:46.315021 15667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39729 every 8 connection(s)
I20260926 23:26:46.316514 15540 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:26:46.320238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15540
I20260926 23:26:46.321134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35275
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37639
--webserver_port=35105
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:26:46.324290 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:46.320641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:45011)
0926 23:26:46.322771 (+ 2130us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:46.323021 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":741,"threads_started":1}
W20260926 23:26:46.325261 15668 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37639 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37639: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
I20260926 23:26:46.326327 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:46.325850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:43139)
0926 23:26:46.326057 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:46.326159 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:26:46.327296 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:46.326856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:37987)
0926 23:26:46.327030 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:46.327124 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:26:46.327564 15668 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:26:46.648244 15671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:26:46.648501 15671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:26:46.648593 15671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:26:46.655766 15671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:26:46.655885 15671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:26:46.655999 15671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:26:46.665884 15671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35105
--tserver_master_addrs=127.0.97.62:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:26:46.667284 15671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:26:46.669121 15671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:26:46.683050 15676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:46.683133 15679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:26:46.683082 15677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:26:46.684039 15671 server_base.cc:1034] running on GCE node
I20260926 23:26:46.684823 15671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:26:46.685889 15671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:26:46.687126 15671 hybrid_clock.cc:648] HybridClock initialized: now 1790465206687072 us; error 48 us; skew 500 ppm
I20260926 23:26:46.687467 15671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:26:46 dist-test-slave-q5g8 krb5kdc[14227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:26:46.690798 15671 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:26:46.692345 15671 webserver.cc:460] Webserver started at http://127.0.97.3:35105/ using document root <none> and password file <none>
I20260926 23:26:46.693181 15671 fs_manager.cc:359] Metadata directory not provided
I20260926 23:26:46.693342 15671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:26:46.699126 15671 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:26:46.702167 15686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:46.703809 15671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:26:46.704020 15671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "243d62011ce5442987cf11d2bf203785"
format_stamp: "Formatted at 2026-09-26 23:26:41 on dist-test-slave-q5g8"
server_key: "72efe5cbe5f58c62af7976c32110be7f"
server_key_iv: "f73d3af26614efda0b1c10cebb8222eb"
server_key_version: "encryptionkey@0"
I20260926 23:26:46.705150 15671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:26:46.723348 15671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:26:46.727574 15671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:26:46.727873 15671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:26:46.728931 15671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:26:46.730281 15671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:26:46.730417 15671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:46.730543 15671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:26:46.730629 15671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:26:46.765391 15671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35275
I20260926 23:26:46.765487 15798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35275 every 8 connection(s)
I20260926 23:26:46.766986 15671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:26:46.771832 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:46.769998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:59193)
0926 23:26:46.770727 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:46.770964 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":292,"threads_started":1}
W20260926 23:26:46.772541 15799 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37639 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37639: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
I20260926 23:26:46.773475 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:46.773131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:55833)
0926 23:26:46.773289 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:46.773358 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:26:46.774369 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:46.774056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:44047)
0926 23:26:46.774179 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:46.774262 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:26:46.774688 15799 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:26:46.775926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15671
I20260926 23:26:46.862195 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:46.861439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:57093)
0926 23:26:46.861880 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:46.861967 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":207,"threads_started":1}
I20260926 23:26:47.329296 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:47.328226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:48089)
0926 23:26:47.328858 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:47.329001 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":241,"threads_started":1}
I20260926 23:26:47.776269 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:47.775391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:57931)
0926 23:26:47.775918 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:47.776024 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":285,"threads_started":1}
I20260926 23:26:47.863775 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:47.863077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:44673)
0926 23:26:47.863533 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:47.863614 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":228,"threads_started":1}
I20260926 23:26:48.330955 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:48.330172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:36735)
0926 23:26:48.330688 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:48.330778 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
I20260926 23:26:48.777842 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:48.777145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:55155)
0926 23:26:48.777594 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:48.777681 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260926 23:26:48.865443 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:48.864729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:39191)
0926 23:26:48.865205 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:48.865279 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":220,"threads_started":1}
I20260926 23:26:49.332978 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:49.332033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:57703)
0926 23:26:49.332594 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:49.332765 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":294,"threads_started":1}
I20260926 23:26:49.779500 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:49.778772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:48147)
0926 23:26:49.779233 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:49.779320 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260926 23:26:49.867430 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:49.866410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:35791)
0926 23:26:49.867072 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:49.867158 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":253,"threads_started":1}
I20260926 23:26:50.334811 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:50.334021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:54771)
0926 23:26:50.334537 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:50.334629 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":247,"threads_started":1}
I20260926 23:26:50.781348 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:50.780487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:51083)
0926 23:26:50.781057 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:50.781147 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":329,"threads_started":1}
I20260926 23:26:50.869575 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:50.868521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:50309)
0926 23:26:50.869164 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:50.869295 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":337,"threads_started":1}
I20260926 23:26:51.337077 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:51.336083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:51617)
0926 23:26:51.336654 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:51.336838 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":310,"threads_started":1}
I20260926 23:26:51.783558 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:51.782618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:54127)
0926 23:26:51.783271 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:51.783364 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":375,"threads_started":1}
I20260926 23:26:51.871542 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:51.870654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:57107)
0926 23:26:51.871230 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:51.871361 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":347,"threads_started":1}
I20260926 23:26:52.339079 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:52.338233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:47371)
0926 23:26:52.338776 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:52.338882 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":303,"threads_started":1}
I20260926 23:26:52.785383 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:52.784537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:59399)
0926 23:26:52.785071 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:52.785214 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":302,"threads_started":1}
I20260926 23:26:52.873723 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:52.872629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:59019)
0926 23:26:52.873327 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:52.873420 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":289,"threads_started":1}
I20260926 23:26:53.341493 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:53.340373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:37591)
0926 23:26:53.341134 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:53.341239 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":411,"threads_started":1}
I20260926 23:26:53.787248 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:53.786415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:36009)
0926 23:26:53.786928 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:53.787014 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":221,"threads_started":1}
I20260926 23:26:53.875490 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:53.874699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:41653)
0926 23:26:53.875235 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:53.875312 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":333,"threads_started":1}
I20260926 23:26:54.343307 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:54.342508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:59043)
0926 23:26:54.343021 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:54.343131 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260926 23:26:54.789100 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:54.788274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:49355)
0926 23:26:54.788829 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:54.788921 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":268,"threads_started":1}
I20260926 23:26:54.877405 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:54.876546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:55357)
0926 23:26:54.877124 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:54.877218 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":280,"threads_started":1}
I20260926 23:26:55.345147 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:55.344263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:44707)
0926 23:26:55.344849 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:55.344941 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":281,"threads_started":1}
I20260926 23:26:55.790974 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:55.790085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:52383)
0926 23:26:55.790682 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:55.790780 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
I20260926 23:26:55.879191 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:55.878436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:40933)
0926 23:26:55.878902 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:55.878987 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":253,"threads_started":1}
I20260926 23:26:56.346930 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:56.346198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:47213)
0926 23:26:56.346682 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:56.346767 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":278,"threads_started":1}
I20260926 23:26:56.792830 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:56.792031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:50495)
0926 23:26:56.792499 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:56.792586 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":256,"threads_started":1}
I20260926 23:26:56.881042 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:56.880254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:34991)
0926 23:26:56.880756 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:56.880855 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":227,"threads_started":1}
I20260926 23:26:57.348985 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:57.347994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:37773)
0926 23:26:57.348575 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:57.348715 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":290,"threads_started":1}
I20260926 23:26:57.794730 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:57.793883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:53761)
0926 23:26:57.794389 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:57.794531 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":264,"threads_started":1}
I20260926 23:26:57.882670 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:57.881977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:60153)
0926 23:26:57.882418 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:57.882514 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260926 23:26:58.350871 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:58.350032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:48441)
0926 23:26:58.350602 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:58.350696 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":304,"threads_started":1}
I20260926 23:26:58.796572 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:58.795819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:32987)
0926 23:26:58.796300 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:58.796390 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260926 23:26:58.884598 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:58.883713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:60635)
0926 23:26:58.884278 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:58.884392 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":256,"threads_started":1}
I20260926 23:26:59.352924 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:59.352055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:33727)
0926 23:26:59.352604 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:59.352726 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":282,"threads_started":1}
I20260926 23:26:59.798421 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:59.797626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:39527)
0926 23:26:59.798112 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:59.798214 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":234,"threads_started":1}
I20260926 23:26:59.886560 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:26:59.885700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:58473)
0926 23:26:59.886277 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 23:26:59.886368 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":372,"threads_started":1}
I20260926 23:27:00.354947 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:00.354145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:38649)
0926 23:27:00.354665 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:00.354759 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260926 23:27:00.800339 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:00.799482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:35679)
0926 23:27:00.799982 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:00.800136 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":263,"threads_started":1}
I20260926 23:27:00.888942 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:00.887897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.1:57399)
0926 23:27:00.888481 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:00.888601 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":303,"threads_started":1}
I20260926 23:27:01.357193 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:01.356279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.2:53557)
0926 23:27:01.356918 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:01.357013 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":327,"threads_started":1}
I20260926 23:27:01.781250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 15374
I20260926 23:27:01.802850 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:01.801787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37639 (local address 127.0.97.3:40271)
0926 23:27:01.802454 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:01.802579 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":298,"threads_started":1}
I20260926 23:27:01.803592 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:01.782739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.0.97.1:41807)
0926 23:27:01.783841 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:01.783859 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:01.783902 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:27:01.786093 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:01.786104 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:01.786624 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:01.787051 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:01.788862 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:01.790231 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:01.791653 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:01.792434 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:01.792672 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:01.795676 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:01.795685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:01.795704 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:01.795753 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:01.799281 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:01.800184 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:01.800194 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:01.800202 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:01.800411 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:01.800933 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:01.800938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:01.800943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:01.801766 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:01.802190 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:01.802693 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 23:27:01.803259 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":684,"threads_started":1}
I20260926 23:27:01.805361 15532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:01.896417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 15374
I20260926 23:27:01.914469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 15540
I20260926 23:27:01.933993 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:01.915558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.97.2:39729)
0926 23:27:01.916179 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:01.916194 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:01.916233 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:27:01.917890 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:01.917902 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:01.918376 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:01.918703 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:01.920078 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:01.921581 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:01.922831 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:01.923609 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:01.923760 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:01.926527 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:01.926536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:01.926556 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:01.926621 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:01.930272 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:01.931093 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:01.931103 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:01.931114 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:01.931324 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:01.931780 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:01.931783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:01.931785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:01.932422 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:01.932785 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:01.933201 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 23:27:01.933639 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":317,"threads_started":1}
I20260926 23:27:01.935598 15663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:02.025147 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 15540
I20260926 23:27:02.045708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 15671
I20260926 23:27:02.067989 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:02.046838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.0.97.3:35275)
0926 23:27:02.047453 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:02.047470 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:02.047513 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:27:02.049987 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:02.050004 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:02.050701 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:02.051267 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:02.052907 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:02.054252 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:02.055780 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:02.056616 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:02.056926 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:02.060049 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:02.060058 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:02.060078 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:02.060133 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:02.063839 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:02.064924 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:02.064934 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:02.064943 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:02.065221 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:02.065683 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:02.065687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:02.065689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:02.066412 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:02.066696 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:02.067269 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 23:27:02.067659 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260926 23:27:02.069633 15794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:02.159109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 15671
2026-09-26T23:27:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24533 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:27:02 dist-test-slave-q5g8 krb5kdc[15857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:27:02 dist-test-slave-q5g8 krb5kdc[15857](info): set up 2 sockets
Sep 26 23:27:02 dist-test-slave-q5g8 krb5kdc[15857](info): commencing operation
krb5kdc: starting...
W20260926 23:27:03.358335 388 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:27:03 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:27:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:27:03.634493 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:03.988632 15870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:03.989027 15870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:03.989241 15870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:03.997144 15870 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:27:03.997263 15870 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:27:03.997354 15870 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:27:03.997437 15870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:27:03.997546 15870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:03.997638 15870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:03.997725 15870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:27:03.997815 15870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:27:04.008164 15870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:04.009582 15870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:04.011023 15870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:04.024058 15875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:04.024084 15876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:04.024142 15878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:04.024751 15870 server_base.cc:1034] running on GCE node
I20260926 23:27:04.025627 15870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:04.027137 15870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:04.028369 15870 hybrid_clock.cc:648] HybridClock initialized: now 1790465224028312 us; error 31 us; skew 500 ppm
I20260926 23:27:04.028786 15870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:04 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:04.039011 15870 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:27:04.040920 15870 webserver.cc:460] Webserver started at http://127.0.97.62:38901/ using document root <none> and password file <none>
I20260926 23:27:04.041783 15870 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:04.041887 15870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:04.042279 15870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:27:04.045879 15870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "158c634d308041ac8440a3c53985b275"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "f6c0b3bf360f750f2d50e7538cc2d393"
server_key_iv: "a4ccfb4edad1366e3e4b13f3562a3f39"
server_key_version: "encryptionkey@0"
I20260926 23:27:04.046880 15870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "158c634d308041ac8440a3c53985b275"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "f6c0b3bf360f750f2d50e7538cc2d393"
server_key_iv: "a4ccfb4edad1366e3e4b13f3562a3f39"
server_key_version: "encryptionkey@0"
I20260926 23:27:04.053239 15870 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:27:04.057643 15885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:04.059911 15870 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:27:04.060135 15870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "158c634d308041ac8440a3c53985b275"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "f6c0b3bf360f750f2d50e7538cc2d393"
server_key_iv: "a4ccfb4edad1366e3e4b13f3562a3f39"
server_key_version: "encryptionkey@0"
I20260926 23:27:04.060397 15870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:04.080163 15870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:04.085287 15870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:04.085610 15870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:04.107474 15870 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45365
I20260926 23:27:04.107494 15936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45365 every 8 connection(s)
I20260926 23:27:04.109115 15870 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:27:04.110812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15870
I20260926 23:27:04.111084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:27:04.111809 388 external_mini_cluster.cc:1428] Setting key dcea99951c255f25077acd79a6e8f9b9
I20260926 23:27:04.114815 15937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:27:04.127873 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Bootstrap starting.
Sep 26 23:27:04 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:27:04.132138 15937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Neither blocks nor log segments found. Creating new log.
I20260926 23:27:04.133606 15937 log.cc:824] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:04.137989 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: No bootstrap required, opened a new log
I20260926 23:27:04.139726 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:04.114323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.0.97.62:45365)
0926 23:27:04.115401 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:04.115416 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:04.115466 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:27:04.117513 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:04.117542 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:04.118076 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:04.118649 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:04.120494 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:04.122141 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:04.123766 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:04.124649 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:04.124967 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:04.129713 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:04.129723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:04.129741 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:04.129803 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:04.133692 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:04.134946 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:04.134955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:04.134963 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:04.135397 (+ 434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:04.135914 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:04.135919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:04.135923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:04.136790 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:04.137205 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:04.137589 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 23:27:04.138137 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":483,"threads_started":1}
I20260926 23:27:04.148180 15937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:04.148798 15937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:27:04.148923 15937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158c634d308041ac8440a3c53985b275, State: Initialized, Role: FOLLOWER
I20260926 23:27:04.149806 15937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:04.150166 15937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:27:04.150359 15937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:27:04.150604 15937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:27:04.153601 15937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:04.154254 15937 leader_election.cc:302] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 158c634d308041ac8440a3c53985b275; no voters:
I20260926 23:27:04.155109 15937 leader_election.cc:288] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:27:04.155301 15942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:27:04.156347 15942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 LEADER]: Becoming Leader. State: Replica: 158c634d308041ac8440a3c53985b275, State: Running, Role: LEADER
I20260926 23:27:04.157096 15942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:04.157971 15937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:27:04.160640 15944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158c634d308041ac8440a3c53985b275. Latest consensus state: current_term: 1 leader_uuid: "158c634d308041ac8440a3c53985b275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260926 23:27:04.161144 15944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:27:04.160632 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "158c634d308041ac8440a3c53985b275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260926 23:27:04.162043 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:27:04.174391 15950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:27:04.181198 15950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:27:04.192178 15950 catalog_manager.cc:1347] Generated new cluster ID: 8a6c35a3732b424f916f0f183787b5da
I20260926 23:27:04.192322 15950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:27:04.202126 15950 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:27:04.203564 15950 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:27:04.211100 15950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Generated new TSK 0
I20260926 23:27:04.212137 15950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:27:04.293514 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:04.669667 15963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:04.669970 15963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:04.670125 15963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:04.678066 15963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:04.678203 15963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:04.678417 15963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:27:04.689148 15963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:04.690536 15963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:04.691886 15963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:04.706537 15971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:04.706609 15968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:04.706650 15969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:04.707271 15963 server_base.cc:1034] running on GCE node
I20260926 23:27:04.707908 15963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:04.708950 15963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:04.710157 15963 hybrid_clock.cc:648] HybridClock initialized: now 1790465224710102 us; error 46 us; skew 500 ppm
I20260926 23:27:04.710523 15963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:04 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:04.714125 15963 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:27:04.715719 15963 webserver.cc:460] Webserver started at http://127.0.97.1:36201/ using document root <none> and password file <none>
I20260926 23:27:04.716430 15963 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:04.716538 15963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:04.716922 15963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:27:04.720065 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "17ae87cdee2e476e8e84134920028488"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "b63adbd5eeec82028266310df4686fb6"
server_key_iv: "67309d01533ad25f486c74f2747d40ae"
server_key_version: "encryptionkey@0"
I20260926 23:27:04.720948 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "17ae87cdee2e476e8e84134920028488"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "b63adbd5eeec82028266310df4686fb6"
server_key_iv: "67309d01533ad25f486c74f2747d40ae"
server_key_version: "encryptionkey@0"
I20260926 23:27:04.726405 15963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:27:04.730159 15978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:04.731753 15963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:27:04.731971 15963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "17ae87cdee2e476e8e84134920028488"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "b63adbd5eeec82028266310df4686fb6"
server_key_iv: "67309d01533ad25f486c74f2747d40ae"
server_key_version: "encryptionkey@0"
I20260926 23:27:04.732218 15963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:04.748667 15963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:04.753705 15963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:04.754051 15963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:04.755101 15963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:04.756524 15963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:04.756603 15963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:04.756807 15963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:04.756881 15963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:04.793221 15963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41129
I20260926 23:27:04.793246 16090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41129 every 8 connection(s)
I20260926 23:27:04.794687 15963 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:27:04.801245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 15963
I20260926 23:27:04.801517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:27:04.802400 388 external_mini_cluster.cc:1428] Setting key 9c10f1ffc4c6a828a84c1b27de42459c
Sep 26 23:27:04 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:27:04.818516 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:04.797592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46483 (local address 127.0.97.62:45365)
0926 23:27:04.798073 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:04.798078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:04.799237 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 23:27:04.801500 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:04.801504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:04.801602 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:04.801729 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:04.803897 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:04.804739 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:04.806430 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:04.806844 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:04.806990 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:04.810980 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:04.810984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:04.810988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:04.811023 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:04.814448 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:04.815277 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:04.815286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:04.815292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:04.815455 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:04.815981 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:04.815989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:04.815995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:04.816437 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:04.816601 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:04.817730 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0926 23:27:04.818013 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":223,"threads_started":1}
I20260926 23:27:04.819347 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:04.798229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:46483)
0926 23:27:04.799012 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:04.799055 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:04.801115 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:04.802046 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:04.802058 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:04.802899 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:27:04.803712 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:04.803734 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:04.804911 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:04.804915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:04.806285 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:04.806297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:04.807108 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:04.807115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:04.807374 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:04.808154 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:27:04.808180 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:27:04.810738 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:27:04.814630 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:04.814636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:04.814651 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:04.815139 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:04.815605 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:04.815610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:04.815613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:04.815851 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:04.816607 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:27:04.816613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:27:04.817224 (+ 611us) client_negotiation.cc:769] Sending connection context
0926 23:27:04.817608 (+ 384us) client_negotiation.cc:240] Negotiation successful
0926 23:27:04.818037 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":313,"threads_started":1}
I20260926 23:27:04.821120 16091 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260926 23:27:04.821488 16091 heartbeater.cc:464] Registering TS with master...
I20260926 23:27:04.822459 16091 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260926 23:27:04.824949 15902 ts_manager.cc:194] Registered new tserver with Master: 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129)
I20260926 23:27:04.826453 15902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46483
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:27:04.860641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:05.212020 16097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:05.212339 16097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:05.212512 16097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:05.219959 16097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:05.220139 16097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:05.220350 16097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:27:05.230794 16097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:05.232272 16097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:05.233753 16097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:05.248517 16102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:05.248546 16105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:05.248453 16103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:05.249500 16097 server_base.cc:1034] running on GCE node
I20260926 23:27:05.250221 16097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:05.251353 16097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:05.252626 16097 hybrid_clock.cc:648] HybridClock initialized: now 1790465225252562 us; error 48 us; skew 500 ppm
I20260926 23:27:05.253021 16097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:05 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:05.256647 16097 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:27:05.258507 16097 webserver.cc:460] Webserver started at http://127.0.97.2:40285/ using document root <none> and password file <none>
I20260926 23:27:05.259351 16097 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:05.259510 16097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:05.259860 16097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:27:05.263203 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "2d88fef63af4453cbbd4189b48942199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "d8d69629e43026f3ee20a9ae94d7d10d"
server_key_iv: "fb091a5f18a8ebf059f948841c876201"
server_key_version: "encryptionkey@0"
I20260926 23:27:05.264223 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2d88fef63af4453cbbd4189b48942199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "d8d69629e43026f3ee20a9ae94d7d10d"
server_key_iv: "fb091a5f18a8ebf059f948841c876201"
server_key_version: "encryptionkey@0"
I20260926 23:27:05.270115 16097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 23:27:05.274178 16112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:05.275844 16097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260926 23:27:05.276053 16097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "2d88fef63af4453cbbd4189b48942199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "d8d69629e43026f3ee20a9ae94d7d10d"
server_key_iv: "fb091a5f18a8ebf059f948841c876201"
server_key_version: "encryptionkey@0"
I20260926 23:27:05.276309 16097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:05.301403 16097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:05.305892 16097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:05.306232 16097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:05.307338 16097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:05.308940 16097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:05.309051 16097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:05.309221 16097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:05.309298 16097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:05.344821 16097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38729
I20260926 23:27:05.344864 16224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38729 every 8 connection(s)
I20260926 23:27:05.346170 16097 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:27:05.355832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 16097
I20260926 23:27:05.356042 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:27:05.356639 388 external_mini_cluster.cc:1428] Setting key f2fcbc03ce1a0cd9c40a8384befdfb27
Sep 26 23:27:05 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:27:05.370353 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.348776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40503 (local address 127.0.97.62:45365)
0926 23:27:05.349208 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:05.349213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:05.350312 (+ 1099us) server_negotiation.cc:407] Connection header received
0926 23:27:05.352259 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:05.352264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:05.352368 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:05.352491 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:05.354180 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.354927 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.357512 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.358101 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.358246 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:05.362690 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:05.362699 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:05.362707 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:05.362757 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:05.366753 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:05.367618 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:05.367626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:05.367633 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:05.367810 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:05.368392 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:05.368395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:05.368397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:05.368740 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:05.368870 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:05.369855 (+ 985us) server_negotiation.cc:299] Negotiation successful
0926 23:27:05.370045 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260926 23:27:05.371991 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.349365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:40503)
0926 23:27:05.350091 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:05.350138 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:05.351941 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:05.352779 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:05.352789 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:05.353376 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:27:05.353978 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:05.353997 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.355106 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.355111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:05.357343 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:05.357357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.358365 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.358372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:05.358649 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:05.359497 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:27:05.359530 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:27:05.362386 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:27:05.366931 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:05.366939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:05.366955 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:05.367461 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:05.367934 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:05.367941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:05.367947 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:05.368265 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:05.368882 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:27:05.368893 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:27:05.369431 (+ 538us) client_negotiation.cc:769] Sending connection context
0926 23:27:05.369751 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 23:27:05.370394 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":284,"threads_started":1}
I20260926 23:27:05.373636 16225 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260926 23:27:05.374017 16225 heartbeater.cc:464] Registering TS with master...
I20260926 23:27:05.374975 16225 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260926 23:27:05.376660 15902 ts_manager.cc:194] Registered new tserver with Master: 2d88fef63af4453cbbd4189b48942199 (127.0.97.2:38729)
I20260926 23:27:05.377660 15902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40503
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:27:05.416302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:05.775578 16231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:05.775903 16231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:05.776000 16231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:05.783995 16231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:05.784140 16231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:05.784267 16231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:27:05.794731 16231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:05.796233 16231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:05.797740 16231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:05.812381 16239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:05.812409 16236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:05.812435 16237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:05.813429 16231 server_base.cc:1034] running on GCE node
I20260926 23:27:05.814146 16231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:05.815279 16231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:05.816515 16231 hybrid_clock.cc:648] HybridClock initialized: now 1790465225816464 us; error 45 us; skew 500 ppm
I20260926 23:27:05.816957 16231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:05 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:05.820832 16231 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:27:05.822638 16231 webserver.cc:460] Webserver started at http://127.0.97.3:37337/ using document root <none> and password file <none>
I20260926 23:27:05.823452 16231 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:05.823629 16231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:05.823974 16231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:27:05.827234 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "2118d46059674c61acdbf973137de199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "4691e023bf352c21b0d91d6ae8ba8a2b"
server_key_iv: "647ff2c25d135e4d9b3372df3b0d4e24"
server_key_version: "encryptionkey@0"
I20260926 23:27:05.828078 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2118d46059674c61acdbf973137de199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "4691e023bf352c21b0d91d6ae8ba8a2b"
server_key_iv: "647ff2c25d135e4d9b3372df3b0d4e24"
server_key_version: "encryptionkey@0"
I20260926 23:27:05.830120 16091 heartbeater.cc:503] Master 127.0.97.62:45365 was elected leader, sending a full tablet report...
I20260926 23:27:05.834230 16231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:27:05.838141 16246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:05.839705 16231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:27:05.839936 16231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "2118d46059674c61acdbf973137de199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "4691e023bf352c21b0d91d6ae8ba8a2b"
server_key_iv: "647ff2c25d135e4d9b3372df3b0d4e24"
server_key_version: "encryptionkey@0"
I20260926 23:27:05.840209 16231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:05.856577 16231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:05.861239 16231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:05.861560 16231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:05.862564 16231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:05.863957 16231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:05.864064 16231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:05.864256 16231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:05.864324 16231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:05.900969 16231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35609
I20260926 23:27:05.900991 16358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35609 every 8 connection(s)
I20260926 23:27:05.902401 16231 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:27:05.911357 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 16231
I20260926 23:27:05.911561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:27:05.912057 388 external_mini_cluster.cc:1428] Setting key 6cbbca09951f060b9af33740c290a001
Sep 26 23:27:05 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:27:05.924507 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.905285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44519 (local address 127.0.97.62:45365)
0926 23:27:05.905751 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:05.905755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:05.906872 (+ 1117us) server_negotiation.cc:407] Connection header received
0926 23:27:05.908957 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:05.908964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:05.909074 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:05.909207 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:05.910914 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.911851 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.913911 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.914491 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.914650 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:05.918557 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:05.918564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:05.918568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:05.918607 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:05.921352 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:05.922109 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:05.922115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:05.922118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:05.922248 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:05.922701 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:05.922706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:05.922714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:05.922989 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:05.923149 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:05.923873 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 23:27:05.924110 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":210,"threads_started":1}
I20260926 23:27:05.925508 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.905856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:44519)
0926 23:27:05.906690 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:05.906740 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:05.908593 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:05.909468 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:05.909477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:05.910108 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:27:05.910724 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:05.910742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.912028 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.912039 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:05.913734 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:05.913748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.914752 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.914760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:05.915044 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:05.915887 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:27:05.915914 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:27:05.918306 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:27:05.921501 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:05.921510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:05.921529 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:05.921968 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:05.922366 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:05.922372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:05.922376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:05.922575 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:05.923119 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:27:05.923127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:27:05.923480 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 23:27:05.923757 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 23:27:05.924320 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260926 23:27:05.927105 16359 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260926 23:27:05.927451 16359 heartbeater.cc:464] Registering TS with master...
I20260926 23:27:05.928390 16359 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260926 23:27:05.929857 15902 ts_manager.cc:194] Registered new tserver with Master: 2118d46059674c61acdbf973137de199 (127.0.97.3:35609)
I20260926 23:27:05.930637 15902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44519
I20260926 23:27:05.940964 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:27:05.960181 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.946250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.97.62:45365)
0926 23:27:05.946545 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:05.946554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:05.946887 (+ 333us) server_negotiation.cc:407] Connection header received
0926 23:27:05.947122 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:05.947127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:05.947199 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:05.947316 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:05.949143 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.949881 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.951455 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:05.951879 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:05.952022 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:05.954667 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:05.954672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:05.954676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:05.954709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:05.957232 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:05.957959 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:05.957971 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:05.957975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:05.958117 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:05.958568 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:05.958573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:05.958578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:05.958934 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:05.959225 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:05.959579 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 23:27:05.959837 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 23:27:05.964530 15902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:27:05.971817 15902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:27:06.007632 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.986803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.0.97.3:35609)
0926 23:27:05.998287 (+ 11484us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:05.998300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:05.998345 (+ 45us) server_negotiation.cc:407] Connection header received
0926 23:27:05.998476 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:05.998489 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:05.998856 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:05.999091 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:06.000903 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.002494 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.004993 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.006299 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.006463 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.006693 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.007042 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.007210 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11267,"thread_start_us":11203,"threads_started":1}
I20260926 23:27:06.007635 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.986107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.0.97.1:41129)
0926 23:27:05.993324 (+ 7217us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:05.993341 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:05.997971 (+ 4630us) server_negotiation.cc:407] Connection header received
0926 23:27:05.998547 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:05.998556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:05.998874 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:05.999091 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:06.000575 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.002234 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.004654 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.006313 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.006421 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.006667 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.007101 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.007271 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":560,"threads_started":1}
I20260926 23:27:06.007709 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.985773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.0.1:43692)
0926 23:27:05.997780 (+ 12007us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:05.997829 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:05.998332 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:05.999511 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:05.999519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:05.999567 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:06.000388 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.000399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.002390 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.002395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.004452 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.004473 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.006798 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.006802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.006921 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.006942 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 23:27:06.007052 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 23:27:06.007209 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11695,"mutex_wait_us":35,"thread_start_us":11710,"threads_started":1}
I20260926 23:27:06.007814 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.986205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35609 (local address 127.0.0.1:59474)
0926 23:27:05.997824 (+ 11619us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:05.997852 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:05.998083 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:05.999419 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:05.999426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:05.999467 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:06.000217 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.000233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.002658 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.002664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.004801 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.004817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.006510 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.006516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.006796 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.006838 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:27:06.006988 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 23:27:06.007209 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11460,"mutex_wait_us":75,"spinlock_wait_cycles":3328,"thread_start_us":13762,"threads_started":1}
I20260926 23:27:06.011683 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.987478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.97.2:38729)
0926 23:27:05.999277 (+ 11799us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:05.999285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:06.000495 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 23:27:06.000822 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:06.000833 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:06.001098 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:06.001328 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:06.002135 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.003694 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.009026 (+ 5332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.010587 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.010731 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.010941 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.011210 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.011384 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11621,"thread_start_us":11797,"threads_started":1}
I20260926 23:27:06.011691 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:05.987253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38729 (local address 127.0.0.1:51944)
0926 23:27:06.000377 (+ 13124us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:06.000398 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:06.000601 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:06.001567 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:06.001572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:06.001597 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:06.001964 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.001976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.003888 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.003894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.008811 (+ 4917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.008827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.010819 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.010826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.010999 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.011035 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:27:06.011147 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 23:27:06.011302 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12945,"mutex_wait_us":1615,"thread_start_us":10004,"threads_started":1}
I20260926 23:27:06.012132 16026 tablet_service.cc:1467] Processing CreateTablet for tablet f580f02988054e9cb2c41bead55a1702 (DEFAULT_TABLE table=test-table [id=37c700525a5740869761cd6cdc22ae0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:27:06.012125 16294 tablet_service.cc:1467] Processing CreateTablet for tablet f580f02988054e9cb2c41bead55a1702 (DEFAULT_TABLE table=test-table [id=37c700525a5740869761cd6cdc22ae0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:27:06.013597 16294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f580f02988054e9cb2c41bead55a1702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:27:06.013664 16026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f580f02988054e9cb2c41bead55a1702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:27:06.015908 16160 tablet_service.cc:1467] Processing CreateTablet for tablet f580f02988054e9cb2c41bead55a1702 (DEFAULT_TABLE table=test-table [id=37c700525a5740869761cd6cdc22ae0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:27:06.017441 16160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f580f02988054e9cb2c41bead55a1702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:27:06.026470 16378 tablet_bootstrap.cc:492] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Bootstrap starting.
I20260926 23:27:06.030161 16378 tablet_bootstrap.cc:654] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Neither blocks nor log segments found. Creating new log.
I20260926 23:27:06.031337 16378 log.cc:824] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:06.032438 16379 tablet_bootstrap.cc:492] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Bootstrap starting.
I20260926 23:27:06.034977 16381 tablet_bootstrap.cc:492] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Bootstrap starting.
I20260926 23:27:06.036445 16378 tablet_bootstrap.cc:492] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: No bootstrap required, opened a new log
I20260926 23:27:06.036847 16378 ts_tablet_manager.cc:1397] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260926 23:27:06.036882 16379 tablet_bootstrap.cc:654] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Neither blocks nor log segments found. Creating new log.
I20260926 23:27:06.038359 16379 log.cc:824] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:06.038829 16381 tablet_bootstrap.cc:654] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Neither blocks nor log segments found. Creating new log.
I20260926 23:27:06.039871 16381 log.cc:824] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:06.043181 16381 tablet_bootstrap.cc:492] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: No bootstrap required, opened a new log
I20260926 23:27:06.043553 16381 ts_tablet_manager.cc:1397] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:27:06.048318 16379 tablet_bootstrap.cc:492] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: No bootstrap required, opened a new log
I20260926 23:27:06.048830 16379 ts_tablet_manager.cc:1397] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.013s
I20260926 23:27:06.050812 16378 raft_consensus.cc:348] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.051395 16378 raft_consensus.cc:374] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:27:06.051611 16378 raft_consensus.cc:729] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2118d46059674c61acdbf973137de199, State: Initialized, Role: FOLLOWER
I20260926 23:27:06.052385 16378 consensus_queue.cc:260] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.053787 16359 heartbeater.cc:503] Master 127.0.97.62:45365 was elected leader, sending a full tablet report...
I20260926 23:27:06.053589 16381 raft_consensus.cc:348] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.054076 16381 raft_consensus.cc:374] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:27:06.054193 16381 raft_consensus.cc:729] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d88fef63af4453cbbd4189b48942199, State: Initialized, Role: FOLLOWER
I20260926 23:27:06.054522 16378 ts_tablet_manager.cc:1428] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260926 23:27:06.054864 16381 consensus_queue.cc:260] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.056027 16225 heartbeater.cc:503] Master 127.0.97.62:45365 was elected leader, sending a full tablet report...
I20260926 23:27:06.056778 16381 ts_tablet_manager.cc:1428] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260926 23:27:06.061476 16379 raft_consensus.cc:348] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.062071 16379 raft_consensus.cc:374] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:27:06.062189 16379 raft_consensus.cc:729] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ae87cdee2e476e8e84134920028488, State: Initialized, Role: FOLLOWER
I20260926 23:27:06.062960 16379 consensus_queue.cc:260] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.065073 16379 ts_tablet_manager.cc:1428] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
W20260926 23:27:06.098318 16226 tablet.cc:2369] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:27:06.154129 16360 tablet.cc:2369] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:27:06.297492 16092 tablet.cc:2369] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:27:06.407653 16384 raft_consensus.cc:482] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:27:06.407943 16384 raft_consensus.cc:504] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.409787 16384 leader_election.cc:288] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129), 2d88fef63af4453cbbd4189b48942199 (127.0.97.2:38729)
I20260926 23:27:06.417133 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.410430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53325 (local address 127.0.97.1:41129)
0926 23:27:06.410687 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:06.410697 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:06.410725 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:27:06.410819 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:06.410827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:06.410948 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:06.411092 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:06.411596 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.412648 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.415053 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.416256 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.416408 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.416523 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.416764 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.416908 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 23:27:06.417253 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.410161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.97.3:53325)
0926 23:27:06.410411 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:06.410438 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:06.410580 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:06.411114 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:06.411119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:06.411149 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:06.411396 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.411410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.412797 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.412801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.414856 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.414876 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.416426 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.416430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.416571 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.416597 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 23:27:06.416675 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:27:06.416906 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":71}
I20260926 23:27:06.417788 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.410590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38729 (local address 127.0.97.3:47933)
0926 23:27:06.411212 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:06.411233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:06.411359 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:06.411835 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:06.411838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:06.411858 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:06.412122 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.412128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.413328 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.413334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.415268 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:06.415278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.416916 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.416921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:06.417104 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.417135 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:27:06.417236 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 23:27:06.417376 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":1,"thread_start_us":327,"threads_started":1}
I20260926 23:27:06.418072 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.410535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47933 (local address 127.0.97.2:38729)
0926 23:27:06.410764 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:06.410768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:06.411337 (+ 569us) server_negotiation.cc:407] Connection header received
0926 23:27:06.411610 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:06.411615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:06.411702 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:06.411809 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:06.412288 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.413180 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.415423 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.416681 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.416842 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.416941 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.417662 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.417807 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:27:06.418429 16046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f580f02988054e9cb2c41bead55a1702" candidate_uuid: "2118d46059674c61acdbf973137de199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ae87cdee2e476e8e84134920028488" is_pre_election: true
I20260926 23:27:06.419090 16046 raft_consensus.cc:2454] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2118d46059674c61acdbf973137de199 in term 0.
I20260926 23:27:06.419024 16180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f580f02988054e9cb2c41bead55a1702" candidate_uuid: "2118d46059674c61acdbf973137de199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d88fef63af4453cbbd4189b48942199" is_pre_election: true
I20260926 23:27:06.419601 16180 raft_consensus.cc:2454] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2118d46059674c61acdbf973137de199 in term 0.
I20260926 23:27:06.420055 16247 leader_election.cc:302] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ae87cdee2e476e8e84134920028488, 2118d46059674c61acdbf973137de199; no voters:
I20260926 23:27:06.420617 16384 raft_consensus.cc:2781] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:27:06.420833 16384 raft_consensus.cc:482] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:27:06.420946 16384 raft_consensus.cc:3037] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:27:06.423257 16384 raft_consensus.cc:504] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.423938 16384 leader_election.cc:288] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [CANDIDATE]: Term 1 election: Requested vote from peers 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129), 2d88fef63af4453cbbd4189b48942199 (127.0.97.2:38729)
I20260926 23:27:06.424638 16046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f580f02988054e9cb2c41bead55a1702" candidate_uuid: "2118d46059674c61acdbf973137de199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ae87cdee2e476e8e84134920028488"
I20260926 23:27:06.424603 16180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f580f02988054e9cb2c41bead55a1702" candidate_uuid: "2118d46059674c61acdbf973137de199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d88fef63af4453cbbd4189b48942199"
I20260926 23:27:06.424929 16046 raft_consensus.cc:3037] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:27:06.424944 16180 raft_consensus.cc:3037] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:27:06.427399 16046 raft_consensus.cc:2454] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2118d46059674c61acdbf973137de199 in term 1.
I20260926 23:27:06.427752 16180 raft_consensus.cc:2454] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2118d46059674c61acdbf973137de199 in term 1.
I20260926 23:27:06.427968 16247 leader_election.cc:302] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ae87cdee2e476e8e84134920028488, 2118d46059674c61acdbf973137de199; no voters:
I20260926 23:27:06.428408 16384 raft_consensus.cc:2781] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:27:06.429159 16384 raft_consensus.cc:686] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 1 LEADER]: Becoming Leader. State: Replica: 2118d46059674c61acdbf973137de199, State: Running, Role: LEADER
I20260926 23:27:06.429838 16384 consensus_queue.cc:237] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } }
I20260926 23:27:06.435521 15902 catalog_manager.cc:5495] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2118d46059674c61acdbf973137de199 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2118d46059674c61acdbf973137de199" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } health_report { overall_health: UNKNOWN } } }
I20260926 23:27:06.468487 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.461904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.97.3:35609)
0926 23:27:06.462136 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:06.462141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:06.462674 (+ 533us) server_negotiation.cc:407] Connection header received
0926 23:27:06.462792 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:06.462795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:06.462894 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:06.463045 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:27:06.463489 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.464307 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.466137 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.466581 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.466705 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.467238 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:27:06.467849 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:27:06.467978 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.468179 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.468266 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:27:06.477998 16046 raft_consensus.cc:1261] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Refusing update from remote peer 2118d46059674c61acdbf973137de199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:27:06.477999 16180 raft_consensus.cc:1261] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 1 FOLLOWER]: Refusing update from remote peer 2118d46059674c61acdbf973137de199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:27:06.479215 16384 consensus_queue.cc:1035] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:27:06.479658 16388 consensus_queue.cc:1035] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:27:06.509583 15901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:27:06.509954 15901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:27:06.514323 15901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Sending DeleteTablet for 3 replicas of tablet f580f02988054e9cb2c41bead55a1702
I20260926 23:27:06.515843 16026 tablet_service.cc:1514] Processing DeleteTablet for tablet f580f02988054e9cb2c41bead55a1702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:27:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43692
I20260926 23:27:06.515851 16294 tablet_service.cc:1514] Processing DeleteTablet for tablet f580f02988054e9cb2c41bead55a1702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:27:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59474
I20260926 23:27:06.516206 16160 tablet_service.cc:1514] Processing DeleteTablet for tablet f580f02988054e9cb2c41bead55a1702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:27:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51944
I20260926 23:27:06.517352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 15963
I20260926 23:27:06.528559 16402 tablet_replica.cc:330] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: stopping tablet replica
I20260926 23:27:06.529404 16404 tablet_replica.cc:330] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: stopping tablet replica
I20260926 23:27:06.530267 16404 raft_consensus.cc:2229] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:27:06.530992 16402 raft_consensus.cc:2229] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:27:06.531780 16404 raft_consensus.cc:2258] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:27:06.532783 16403 tablet_replica.cc:330] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: stopping tablet replica
I20260926 23:27:06.533356 16403 raft_consensus.cc:2229] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:27:06.533965 16403 raft_consensus.cc:2258] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:27:06.534145 16402 raft_consensus.cc:2258] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:27:06.536492 16403 ts_tablet_manager.cc:1905] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:27:06.538156 16402 ts_tablet_manager.cc:1905] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:27:06.538662 16404 ts_tablet_manager.cc:1905] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:27:06.545544 16403 ts_tablet_manager.cc:1918] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:27:06.545838 16403 log.cc:1196] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/f580f02988054e9cb2c41bead55a1702
I20260926 23:27:06.546490 16403 ts_tablet_manager.cc:1939] T f580f02988054e9cb2c41bead55a1702 P 17ae87cdee2e476e8e84134920028488: Deleting consensus metadata
I20260926 23:27:06.546994 16404 ts_tablet_manager.cc:1918] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:27:06.547207 16404 log.cc:1196] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/f580f02988054e9cb2c41bead55a1702
I20260926 23:27:06.547878 16404 ts_tablet_manager.cc:1939] T f580f02988054e9cb2c41bead55a1702 P 2d88fef63af4453cbbd4189b48942199: Deleting consensus metadata
I20260926 23:27:06.549085 15886 catalog_manager.cc:4841] TS 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129): tablet f580f02988054e9cb2c41bead55a1702 (table test-table [id=37c700525a5740869761cd6cdc22ae0f]) successfully deleted
Sep 26 23:27:06 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:27:06.550344 15887 catalog_manager.cc:4841] TS 2d88fef63af4453cbbd4189b48942199 (127.0.97.2:38729): tablet f580f02988054e9cb2c41bead55a1702 (table test-table [id=37c700525a5740869761cd6cdc22ae0f]) successfully deleted
I20260926 23:27:06.554420 16402 ts_tablet_manager.cc:1918] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:27:06.554656 16402 log.cc:1196] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/f580f02988054e9cb2c41bead55a1702
I20260926 23:27:06.555307 16402 ts_tablet_manager.cc:1939] T f580f02988054e9cb2c41bead55a1702 P 2118d46059674c61acdbf973137de199: Deleting consensus metadata
I20260926 23:27:06.557369 15889 catalog_manager.cc:4841] TS 2118d46059674c61acdbf973137de199 (127.0.97.3:35609): tablet f580f02988054e9cb2c41bead55a1702 (table test-table [id=37c700525a5740869761cd6cdc22ae0f]) successfully deleted
I20260926 23:27:06.559799 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.518390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.0.97.1:41129)
0926 23:27:06.518637 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:06.518647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:06.529118 (+ 10471us) server_negotiation.cc:407] Connection header received
0926 23:27:06.529509 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:06.529518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:06.529654 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:06.529749 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:06.532521 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.533871 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.536512 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.537293 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.537417 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.551144 (+ 13727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:06.551166 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:06.551188 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:06.551246 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:06.555988 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:06.557219 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:06.557232 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:06.557236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:06.557459 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:06.558103 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:06.558110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:06.558118 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:06.558528 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:06.558779 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.559332 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.559522 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:27:06.560961 16086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:06.654743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 15963
I20260926 23:27:06.674217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 16097
Sep 26 23:27:06 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:27:06.691380 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.675170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.0.97.2:38729)
0926 23:27:06.675367 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:06.675374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:06.675396 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:27:06.675637 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:06.675643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:06.675714 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:06.675841 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:06.677326 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.678041 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.679373 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.679882 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.680022 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.685073 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:06.685089 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:06.685105 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:06.685145 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:06.688250 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:06.689158 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:06.689166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:06.689168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:06.689303 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:06.689728 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:06.689732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:06.689734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:06.690132 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:06.690446 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.690780 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.691018 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:27:06.692111 16220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:06.783749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 16097
I20260926 23:27:06.801402 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 16231
Sep 26 23:27:06 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:27:06.822077 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:06.802320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.0.97.3:35609)
0926 23:27:06.802644 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:06.802652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:06.802701 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:27:06.802786 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:06.802792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:06.802908 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:06.803036 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:06.804456 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.805909 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.807467 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:06.808369 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:06.808533 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:06.813834 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:06.813852 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:06.813875 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:06.813939 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:06.818535 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:06.819410 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:06.819422 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:06.819426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:06.819662 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:06.820087 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:06.820094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:06.820097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:06.820664 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:06.821083 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:06.821342 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 23:27:06.821601 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260926 23:27:06.822952 16354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:06.947628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 16231
I20260926 23:27:06.968565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 15870
I20260926 23:27:06.969805 15932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:07.085608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 15870
I20260926 23:27:27.109691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--webserver_interface=127.0.97.62
--webserver_port=38901
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:27.452173 16410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:27.452538 16410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:27.452759 16410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:27.460227 16410 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:27:27.460345 16410 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:27:27.460417 16410 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:27:27.460526 16410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:27:27.460618 16410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:27.460739 16410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:27.460826 16410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:27:27.460911 16410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:27:27.471055 16410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:27.472472 16410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:27.473872 16410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:27.498158 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:27.498240 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:27.498204 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:27.498588 16410 server_base.cc:1034] running on GCE node
I20260926 23:27:27.499761 16410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:27.501637 16410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:27.502911 16410 hybrid_clock.cc:648] HybridClock initialized: now 1790465247502854 us; error 44 us; skew 500 ppm
I20260926 23:27:27.503338 16410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:27 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:27.512471 16410 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:27:27.514374 16410 webserver.cc:460] Webserver started at http://127.0.97.62:38901/ using document root <none> and password file <none>
I20260926 23:27:27.515247 16410 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:27.515395 16410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:27.522377 16410 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 23:27:27.525605 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:27.527182 16410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:27:27.527390 16410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "158c634d308041ac8440a3c53985b275"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "f6c0b3bf360f750f2d50e7538cc2d393"
server_key_iv: "a4ccfb4edad1366e3e4b13f3562a3f39"
server_key_version: "encryptionkey@0"
I20260926 23:27:27.528483 16410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:27.540534 16410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:27.545264 16410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:27.545593 16410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:27.565944 16410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45365
I20260926 23:27:27.566012 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45365 every 8 connection(s)
I20260926 23:27:27.567576 16410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:27:27.575346 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 16410
I20260926 23:27:27.575770 16478 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:27:27.576500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41129
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45365
--webserver_port=36201
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:27:27.581828 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Bootstrap starting.
I20260926 23:27:27.605365 16478 log.cc:824] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:27.621791 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:27:27.622411 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Bootstrap complete.
I20260926 23:27:27.632907 16478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:27.634009 16478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158c634d308041ac8440a3c53985b275, State: Initialized, Role: FOLLOWER
I20260926 23:27:27.634681 16478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:27.634950 16478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:27:27.635080 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:27:27.635281 16478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:27:27.639729 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:27.640308 16478 leader_election.cc:302] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 158c634d308041ac8440a3c53985b275; no voters:
I20260926 23:27:27.641089 16478 leader_election.cc:288] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:27:27.641250 16482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:27:27.642556 16482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 LEADER]: Becoming Leader. State: Replica: 158c634d308041ac8440a3c53985b275, State: Running, Role: LEADER
I20260926 23:27:27.643267 16482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:27.643673 16478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:27:27.646150 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158c634d308041ac8440a3c53985b275. Latest consensus state: current_term: 2 leader_uuid: "158c634d308041ac8440a3c53985b275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260926 23:27:27.646647 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:27:27.647336 16483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "158c634d308041ac8440a3c53985b275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260926 23:27:27.647663 16483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:27:27.647709 16488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:27:27.659906 16488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:27:27.662133 16488 catalog_manager.cc:1259] Loaded cluster ID: 8a6c35a3732b424f916f0f183787b5da
I20260926 23:27:27.662348 16488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:27:27.668334 16488 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:27:27.670774 16488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Loaded TSK: 0
I20260926 23:27:27.725093 16500 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:27:27.727388 16488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Generated new TSK 1
I20260926 23:27:27.727772 16488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:27:27.963599 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:27.963893 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:27.964036 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:27.971568 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:27.971742 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:27.971915 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:27:27.982213 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36201
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:27.983580 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:27.984987 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:28.000087 16509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:28.000039 16507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:28.000528 16506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:28.000993 16480 server_base.cc:1034] running on GCE node
I20260926 23:27:28.001609 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:28.002571 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:28.003794 16480 hybrid_clock.cc:648] HybridClock initialized: now 1790465248003733 us; error 55 us; skew 500 ppm
I20260926 23:27:28.004110 16480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:28 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:28.007511 16480 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:27:28.009080 16480 webserver.cc:460] Webserver started at http://127.0.97.1:36201/ using document root <none> and password file <none>
I20260926 23:27:28.009773 16480 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:28.009927 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:28.015602 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 23:27:28.018496 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.020006 16480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 23:27:28.020195 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "17ae87cdee2e476e8e84134920028488"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "b63adbd5eeec82028266310df4686fb6"
server_key_iv: "67309d01533ad25f486c74f2747d40ae"
server_key_version: "encryptionkey@0"
I20260926 23:27:28.021242 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:28.039254 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:28.043741 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:28.044052 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:28.045188 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:28.046686 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:28.046797 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.046923 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:28.047005 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.081683 16480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41129
I20260926 23:27:28.081779 16628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41129 every 8 connection(s)
I20260926 23:27:28.083391 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:27:28.086666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 16480
I20260926 23:27:28.087409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38729
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45365
--webserver_port=40285
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:27:28 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:27:28.110925 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:28.086349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44713 (local address 127.0.97.62:45365)
0926 23:27:28.087150 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:28.087172 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:28.088060 (+ 888us) server_negotiation.cc:407] Connection header received
0926 23:27:28.090361 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:28.090382 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:28.090872 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:28.091310 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:28.092747 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.094824 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.097452 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.098107 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.098242 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:28.102705 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:28.102712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:28.102729 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:28.102771 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:28.105987 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.106793 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.106805 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.106816 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.107031 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.107531 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.107535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.107538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.108027 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:28.108238 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:28.109177 (+ 939us) server_negotiation.cc:299] Negotiation successful
0926 23:27:28.109556 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":345,"threads_started":1}
I20260926 23:27:28.110925 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:28.086573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:44713)
0926 23:27:28.087857 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:28.087905 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:28.089749 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:28.091417 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:28.091425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:28.091970 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:27:28.092495 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:28.092510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.095029 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.095036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:28.097312 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:28.097330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.098342 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.098351 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:28.098692 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:28.099528 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:27:28.099563 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:27:28.102404 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:27:28.106145 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:28.106156 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:28.106172 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:28.106638 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:28.107160 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:28.107165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:28.107168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:28.107405 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:28.108169 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:27:28.108177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:27:28.108603 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 23:27:28.108953 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 23:27:28.109522 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":766,"threads_started":1}
I20260926 23:27:28.114394 16629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260926 23:27:28.114708 16629 heartbeater.cc:464] Registering TS with master...
I20260926 23:27:28.115478 16629 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260926 23:27:28.117712 16443 ts_manager.cc:194] Registered new tserver with Master: 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129)
I20260926 23:27:28.119246 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44713
W20260926 23:27:28.421332 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:28.421619 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:28.421794 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:28.429225 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:28.429374 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:28.429533 16633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:27:28.438966 16633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40285
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:28.440196 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:28.441499 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:28.454557 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:28.454582 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:28.454712 16641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:28.455269 16633 server_base.cc:1034] running on GCE node
I20260926 23:27:28.455884 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:28.456842 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:28.458057 16633 hybrid_clock.cc:648] HybridClock initialized: now 1790465248458018 us; error 44 us; skew 500 ppm
I20260926 23:27:28.458369 16633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:28 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:28.461351 16633 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:27:28.462723 16633 webserver.cc:460] Webserver started at http://127.0.97.2:40285/ using document root <none> and password file <none>
I20260926 23:27:28.463387 16633 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:28.463541 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:28.468591 16633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:27:28.471290 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.472553 16633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:27:28.472770 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "2d88fef63af4453cbbd4189b48942199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "d8d69629e43026f3ee20a9ae94d7d10d"
server_key_iv: "fb091a5f18a8ebf059f948841c876201"
server_key_version: "encryptionkey@0"
I20260926 23:27:28.473675 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:28.492599 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:28.496521 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:28.496845 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:28.497800 16633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:28.499109 16633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:28.499229 16633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.499357 16633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:28.499442 16633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.532403 16633 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38729
I20260926 23:27:28.532439 16760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38729 every 8 connection(s)
I20260926 23:27:28.533926 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:27:28.535470 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 16633
I20260926 23:27:28.536355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35609
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45365
--webserver_port=37337
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:27:28 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:27:28.559085 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:28.536474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49193 (local address 127.0.97.62:45365)
0926 23:27:28.536737 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:28.536744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:28.538279 (+ 1535us) server_negotiation.cc:407] Connection header received
0926 23:27:28.541664 (+ 3385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:28.541669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:28.541744 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:28.541846 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:28.544700 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.545907 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.547556 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.548358 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.548509 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:28.551909 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:28.551918 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:28.551926 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:28.551983 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:28.555595 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.556376 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.556384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.556390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.556544 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.557100 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.557108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.557114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.557533 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:28.557707 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:28.558325 (+ 618us) server_negotiation.cc:299] Negotiation successful
0926 23:27:28.558600 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:27:28.559618 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:28.537064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:49193)
0926 23:27:28.538080 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:28.538136 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:28.541297 (+ 3161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:28.542181 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:28.542224 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:28.543367 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:27:28.544427 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:28.544472 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.546120 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.546149 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:28.547408 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:28.547421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.548578 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.548583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:28.548858 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:28.549539 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:27:28.549567 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:27:28.551683 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:27:28.555812 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:28.555818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:28.555830 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:28.556255 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:28.556815 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:28.556818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:28.556820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:28.556978 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:28.557656 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:27:28.557662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:27:28.557956 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 23:27:28.558172 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 23:27:28.558549 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":373,"threads_started":1}
I20260926 23:27:28.561137 16761 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260926 23:27:28.561434 16761 heartbeater.cc:464] Registering TS with master...
I20260926 23:27:28.562129 16761 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260926 23:27:28.563514 16443 ts_manager.cc:194] Registered new tserver with Master: 2d88fef63af4453cbbd4189b48942199 (127.0.97.2:38729)
I20260926 23:27:28.564298 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49193
W20260926 23:27:28.863925 16764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:28.864174 16764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:28.864389 16764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:28.871531 16764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:28.871651 16764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:28.871831 16764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:27:28.881587 16764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37337
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:28.882843 16764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:28.884112 16764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:28.897048 16769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:28.897027 16770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:28.897598 16772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:28.897935 16764 server_base.cc:1034] running on GCE node
I20260926 23:27:28.898480 16764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:28.899391 16764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:28.900589 16764 hybrid_clock.cc:648] HybridClock initialized: now 1790465248900540 us; error 42 us; skew 500 ppm
I20260926 23:27:28.900903 16764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:28 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:28.903935 16764 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:27:28.905323 16764 webserver.cc:460] Webserver started at http://127.0.97.3:37337/ using document root <none> and password file <none>
I20260926 23:27:28.905920 16764 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:28.906025 16764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:28.910902 16764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:27:28.913394 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.914642 16764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:27:28.914801 16764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "2118d46059674c61acdbf973137de199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "4691e023bf352c21b0d91d6ae8ba8a2b"
server_key_iv: "647ff2c25d135e4d9b3372df3b0d4e24"
server_key_version: "encryptionkey@0"
I20260926 23:27:28.915750 16764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:28.930879 16764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:28.935079 16764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:28.935338 16764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:28.936246 16764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:28.937510 16764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:28.937600 16764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.937744 16764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:28.937826 16764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:28.970364 16764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35609
I20260926 23:27:28.970448 16891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35609 every 8 connection(s)
I20260926 23:27:28.971874 16764 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:27:28.973811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 16764
Sep 26 23:27:28 dist-test-slave-q5g8 krb5kdc[15857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:27:28.990285 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:28.975011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.0.97.62:45365)
0926 23:27:28.975763 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:28.975768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:28.975802 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:27:28.975878 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:28.975886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:28.975978 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:28.976107 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:28.977886 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.979017 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.980314 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.981029 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.981187 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:28.984326 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:28.984332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:28.984337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:28.984371 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:28.987353 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.988022 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.988026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.988029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.988147 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.988510 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.988515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.988518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.988916 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:28.989196 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:28.989549 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 23:27:28.989828 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":449,"threads_started":1}
I20260926 23:27:28.994072 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:28.974907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40465 (local address 127.0.97.62:45365)
0926 23:27:28.975160 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:28.975167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:28.976446 (+ 1279us) server_negotiation.cc:407] Connection header received
0926 23:27:28.978743 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:28.978747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:28.978799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:28.978873 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:28.980369 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.981480 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.983028 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.983695 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.983822 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:28.987445 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:28.987451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:28.987455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:28.987504 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:28.990818 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.991703 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.991707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.991710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.991844 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:28.992281 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:28.992286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:28.992289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:28.992610 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:28.992783 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:28.993565 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 23:27:28.993715 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:27:28.994932 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:28.975472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:40465)
0926 23:27:28.976320 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:28.976356 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:28.978446 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:28.979119 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:28.979130 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:28.979659 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:27:28.980182 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:28.980197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.981653 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.981656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:28.982867 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:28.982877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:28.983883 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:28.983887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:28.984074 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:28.984744 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:27:28.984774 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:27:28.987260 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:27:28.990969 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:28.990975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:28.990986 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:28.991574 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:28.991959 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:27:28.991964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:27:28.991967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:27:28.992161 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:27:28.992767 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:27:28.992772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:27:28.993194 (+ 422us) client_negotiation.cc:769] Sending connection context
0926 23:27:28.993470 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 23:27:28.993844 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":318,"threads_started":1}
I20260926 23:27:28.996129 16892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260926 23:27:28.996380 16892 heartbeater.cc:464] Registering TS with master...
I20260926 23:27:28.997079 16892 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260926 23:27:28.998342 16443 ts_manager.cc:194] Registered new tserver with Master: 2118d46059674c61acdbf973137de199 (127.0.97.3:35609)
I20260926 23:27:28.999166 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40465
I20260926 23:27:29.005178 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:27:29.011335 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.005997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.97.62:45365)
0926 23:27:29.006165 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.006173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.006664 (+ 491us) server_negotiation.cc:407] Connection header received
0926 23:27:29.006830 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.006833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.006948 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.007061 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:27:29.007446 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.008214 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.009682 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.010102 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.010195 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.010648 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:27:29.010834 (+ 186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 23:27:29.011101 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59582: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 23:27:29.011459 16631 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59582: token expired
W20260926 23:27:29.011960 16365 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:45365)
I20260926 23:27:29.024840 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.012385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.0.97.62:45365)
0926 23:27:29.012530 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.012534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.012585 (+ 51us) server_negotiation.cc:407] Connection header received
0926 23:27:29.012740 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.012744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.012794 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.012868 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:29.014092 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.014814 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.016068 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.016463 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.016553 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.019663 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:29.019668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:29.019671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:29.019733 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:29.022348 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.022972 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.022975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.022978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.023089 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.023493 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.023497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.023499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.023805 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:29.023979 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.024261 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.024469 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:27:29.028149 16443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:27:29.035804 16443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:27:29.061802 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.047547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.0.1:52804)
0926 23:27:29.048665 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:29.048754 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:29.048973 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:29.054407 (+ 5434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:29.054413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:29.054441 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:29.054953 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.054972 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.056590 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.056596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.058935 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.058952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.060622 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.060627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.060870 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.060942 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 23:27:29.061101 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 23:27:29.061285 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":41,"thread_start_us":3385,"threads_started":1}
I20260926 23:27:29.062326 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.049291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.97.2:38729)
0926 23:27:29.050013 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.050026 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.053457 (+ 3431us) server_negotiation.cc:407] Connection header received
0926 23:27:29.053811 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.053819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.054094 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.054277 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:29.055591 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.057194 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.059722 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.061211 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.061336 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.061525 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.061833 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.061996 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":1212,"threads_started":1}
I20260926 23:27:29.062562 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.047792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.0.97.1:41129)
0926 23:27:29.053910 (+ 6118us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.053921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.053946 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:27:29.054047 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.054052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.054207 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.054406 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:29.055212 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.056465 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.059110 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.060434 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.060571 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.060790 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.062081 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.062239 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5964,"thread_start_us":5843,"threads_started":1}
I20260926 23:27:29.062729 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.049830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.0.97.3:35609)
0926 23:27:29.050661 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.050673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.053766 (+ 3093us) server_negotiation.cc:407] Connection header received
0926 23:27:29.054050 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.054059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.054268 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.054452 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:29.055576 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.057194 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.059460 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.060823 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.060953 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.061194 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.062245 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.062399 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":927,"threads_started":1}
I20260926 23:27:29.062992 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.048357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35609 (local address 127.0.0.1:32910)
0926 23:27:29.053624 (+ 5267us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:29.053644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:29.053871 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:29.054727 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:29.054732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:29.054760 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:29.055317 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.055342 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.057395 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.057398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.059314 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.059327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.061078 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.061087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.061249 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.061281 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:27:29.061366 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 23:27:29.061502 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5114,"mutex_wait_us":9,"thread_start_us":4539,"threads_started":1}
I20260926 23:27:29.063093 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.047772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38729 (local address 127.0.0.1:55696)
0926 23:27:29.053270 (+ 5498us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:29.053294 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:29.053592 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:29.054841 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:29.054847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:29.054886 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:29.055402 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.055413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.057334 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.057339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.059574 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.059587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.061435 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.061444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.061617 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.061642 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:27:29.061775 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 23:27:29.061919 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5316,"mutex_wait_us":1,"spinlock_wait_cycles":12416,"thread_start_us":5545,"threads_started":1}
I20260926 23:27:29.066392 16696 tablet_service.cc:1467] Processing CreateTablet for tablet f473f3ef32cf48b19469f277f12833d2 (DEFAULT_TABLE table=test-table [id=a4902d30994a4e34a223a797c32c605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:27:29.066773 16564 tablet_service.cc:1467] Processing CreateTablet for tablet f473f3ef32cf48b19469f277f12833d2 (DEFAULT_TABLE table=test-table [id=a4902d30994a4e34a223a797c32c605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:27:29.066687 16827 tablet_service.cc:1467] Processing CreateTablet for tablet f473f3ef32cf48b19469f277f12833d2 (DEFAULT_TABLE table=test-table [id=a4902d30994a4e34a223a797c32c605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:27:29.068050 16827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f473f3ef32cf48b19469f277f12833d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:27:29.068053 16564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f473f3ef32cf48b19469f277f12833d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:27:29.067847 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f473f3ef32cf48b19469f277f12833d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:27:29.081264 16908 tablet_bootstrap.cc:492] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Bootstrap starting.
I20260926 23:27:29.084601 16908 tablet_bootstrap.cc:654] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Neither blocks nor log segments found. Creating new log.
I20260926 23:27:29.085906 16908 log.cc:824] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:29.088627 16910 tablet_bootstrap.cc:492] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Bootstrap starting.
I20260926 23:27:29.093649 16910 tablet_bootstrap.cc:654] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Neither blocks nor log segments found. Creating new log.
I20260926 23:27:29.094457 16909 tablet_bootstrap.cc:492] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Bootstrap starting.
I20260926 23:27:29.095233 16910 log.cc:824] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:29.097265 16908 tablet_bootstrap.cc:492] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: No bootstrap required, opened a new log
I20260926 23:27:29.097548 16908 ts_tablet_manager.cc:1397] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20260926 23:27:29.098474 16909 tablet_bootstrap.cc:654] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Neither blocks nor log segments found. Creating new log.
I20260926 23:27:29.099925 16909 log.cc:824] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:29.104821 16910 tablet_bootstrap.cc:492] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: No bootstrap required, opened a new log
I20260926 23:27:29.105084 16910 ts_tablet_manager.cc:1397] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20260926 23:27:29.105113 16909 tablet_bootstrap.cc:492] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: No bootstrap required, opened a new log
I20260926 23:27:29.105348 16909 ts_tablet_manager.cc:1397] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 23:27:29.107455 16908 raft_consensus.cc:348] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.107887 16908 raft_consensus.cc:374] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:27:29.108032 16908 raft_consensus.cc:729] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d88fef63af4453cbbd4189b48942199, State: Initialized, Role: FOLLOWER
I20260926 23:27:29.108661 16908 consensus_queue.cc:260] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.109850 16761 heartbeater.cc:503] Master 127.0.97.62:45365 was elected leader, sending a full tablet report...
I20260926 23:27:29.110445 16908 ts_tablet_manager.cc:1428] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 23:27:29.113752 16909 raft_consensus.cc:348] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.114187 16909 raft_consensus.cc:374] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:27:29.114291 16909 raft_consensus.cc:729] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2118d46059674c61acdbf973137de199, State: Initialized, Role: FOLLOWER
I20260926 23:27:29.114990 16909 consensus_queue.cc:260] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.115275 16910 raft_consensus.cc:348] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.115712 16910 raft_consensus.cc:374] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:27:29.115885 16910 raft_consensus.cc:729] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ae87cdee2e476e8e84134920028488, State: Initialized, Role: FOLLOWER
I20260926 23:27:29.116302 16892 heartbeater.cc:503] Master 127.0.97.62:45365 was elected leader, sending a full tablet report...
I20260926 23:27:29.116505 16910 consensus_queue.cc:260] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.116873 16909 ts_tablet_manager.cc:1428] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 23:27:29.117573 16629 heartbeater.cc:503] Master 127.0.97.62:45365 was elected leader, sending a full tablet report...
I20260926 23:27:29.118173 16910 ts_tablet_manager.cc:1428] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260926 23:27:29.163038 16914 raft_consensus.cc:482] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:27:29.163305 16914 raft_consensus.cc:504] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.164950 16914 leader_election.cc:288] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129), 2118d46059674c61acdbf973137de199 (127.0.97.3:35609)
I20260926 23:27:29.171742 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.165363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35297 (local address 127.0.97.1:41129)
0926 23:27:29.165588 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.165596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.166298 (+ 702us) server_negotiation.cc:407] Connection header received
0926 23:27:29.166470 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.166477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.166559 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.166637 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:29.167295 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.168028 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.170027 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.170779 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.170881 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.170999 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.171350 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.171476 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:27:29.171768 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.165421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59937 (local address 127.0.97.3:35609)
0926 23:27:29.165630 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.165638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.166429 (+ 791us) server_negotiation.cc:407] Connection header received
0926 23:27:29.166578 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.166582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.166686 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.166777 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:27:29.167402 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.168094 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.170035 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.170825 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.170987 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.171086 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.171388 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.171509 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:27:29.171906 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.165306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41129 (local address 127.0.97.2:35297)
0926 23:27:29.166189 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:29.166206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:29.166331 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:29.166680 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:29.166685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:29.166711 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:29.167150 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.167159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.168209 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.168217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.169884 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.169895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.171021 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.171027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.171200 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.171223 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:27:29.171298 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 23:27:29.171435 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":100,"thread_start_us":512,"threads_started":1}
I20260926 23:27:29.172101 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.165388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35609 (local address 127.0.97.2:59937)
0926 23:27:29.166303 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:29.166328 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:27:29.166453 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:27:29.166806 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:27:29.166809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:27:29.166838 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:27:29.167254 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.167265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.168219 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.168223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.169889 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:27:29.169900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.171071 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.171076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:27:29.171220 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.171245 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:27:29.171348 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 23:27:29.171469 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":18,"thread_start_us":455,"threads_started":1}
I20260926 23:27:29.173113 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f473f3ef32cf48b19469f277f12833d2" candidate_uuid: "2d88fef63af4453cbbd4189b48942199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2118d46059674c61acdbf973137de199" is_pre_election: true
I20260926 23:27:29.173103 16584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f473f3ef32cf48b19469f277f12833d2" candidate_uuid: "2d88fef63af4453cbbd4189b48942199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ae87cdee2e476e8e84134920028488" is_pre_election: true
I20260926 23:27:29.173672 16847 raft_consensus.cc:2454] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d88fef63af4453cbbd4189b48942199 in term 0.
I20260926 23:27:29.173672 16584 raft_consensus.cc:2454] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d88fef63af4453cbbd4189b48942199 in term 0.
I20260926 23:27:29.174415 16652 leader_election.cc:302] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17ae87cdee2e476e8e84134920028488, 2118d46059674c61acdbf973137de199, 2d88fef63af4453cbbd4189b48942199; no voters:
I20260926 23:27:29.174865 16914 raft_consensus.cc:2781] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:27:29.175002 16914 raft_consensus.cc:482] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:27:29.175110 16914 raft_consensus.cc:3037] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:27:29.177752 16914 raft_consensus.cc:504] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.178400 16914 leader_election.cc:288] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [CANDIDATE]: Term 1 election: Requested vote from peers 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129), 2118d46059674c61acdbf973137de199 (127.0.97.3:35609)
I20260926 23:27:29.178853 16584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f473f3ef32cf48b19469f277f12833d2" candidate_uuid: "2d88fef63af4453cbbd4189b48942199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ae87cdee2e476e8e84134920028488"
I20260926 23:27:29.178869 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f473f3ef32cf48b19469f277f12833d2" candidate_uuid: "2d88fef63af4453cbbd4189b48942199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2118d46059674c61acdbf973137de199"
I20260926 23:27:29.179062 16847 raft_consensus.cc:3037] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:27:29.179064 16584 raft_consensus.cc:3037] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:27:29.182319 16847 raft_consensus.cc:2454] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d88fef63af4453cbbd4189b48942199 in term 1.
I20260926 23:27:29.182329 16584 raft_consensus.cc:2454] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d88fef63af4453cbbd4189b48942199 in term 1.
I20260926 23:27:29.182727 16652 leader_election.cc:302] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17ae87cdee2e476e8e84134920028488, 2118d46059674c61acdbf973137de199, 2d88fef63af4453cbbd4189b48942199; no voters:
I20260926 23:27:29.183007 16914 raft_consensus.cc:2781] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:27:29.183447 16914 raft_consensus.cc:686] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 1 LEADER]: Becoming Leader. State: Replica: 2d88fef63af4453cbbd4189b48942199, State: Running, Role: LEADER
I20260926 23:27:29.183923 16914 consensus_queue.cc:237] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260926 23:27:29.189018 16442 catalog_manager.cc:5495] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d88fef63af4453cbbd4189b48942199 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2d88fef63af4453cbbd4189b48942199" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d88fef63af4453cbbd4189b48942199" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } health_report { overall_health: UNKNOWN } } }
I20260926 23:27:29.222652 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.215194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.0.97.2:38729)
0926 23:27:29.215416 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.215422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.215442 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:27:29.215523 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.215526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.215628 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.215810 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:27:29.216210 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.217316 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.219557 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.220312 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.220442 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.221200 (+ 758us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:27:29.221952 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:27:29.222118 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.222215 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.222311 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 23:27:29.224131 16893 tablet.cc:2369] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:27:29.232803 16847 raft_consensus.cc:1261] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 1 FOLLOWER]: Refusing update from remote peer 2d88fef63af4453cbbd4189b48942199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:27:29.232928 16584 raft_consensus.cc:1261] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Refusing update from remote peer 2d88fef63af4453cbbd4189b48942199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:27:29.233839 16919 consensus_queue.cc:1035] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2118d46059674c61acdbf973137de199" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:27:29.234633 16919 consensus_queue.cc:1035] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17ae87cdee2e476e8e84134920028488" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:27:29.252482 16924 mvcc.cc:205] Tried to move back new op lower bound from 7333745660845993984 to 7333745660661682176. Current Snapshot: MvccSnapshot[applied={T|T < 7333745660845993984}]
I20260926 23:27:29.269649 16442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:27:29.270174 16442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:27:29.272339 16923 mvcc.cc:205] Tried to move back new op lower bound from 7333745660845993984 to 7333745660661682176. Current Snapshot: MvccSnapshot[applied={T|T < 7333745660845993984}]
I20260926 23:27:29.274756 16442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Sending DeleteTablet for 3 replicas of tablet f473f3ef32cf48b19469f277f12833d2
I20260926 23:27:29.276178 16564 tablet_service.cc:1514] Processing DeleteTablet for tablet f473f3ef32cf48b19469f277f12833d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:27:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52804
I20260926 23:27:29.276194 16826 tablet_service.cc:1514] Processing DeleteTablet for tablet f473f3ef32cf48b19469f277f12833d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:27:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32910
I20260926 23:27:29.276901 16827 tablet_service.cc:1514] Processing DeleteTablet for tablet f473f3ef32cf48b19469f277f12833d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:27:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32910
I20260926 23:27:29.277439 16696 tablet_service.cc:1514] Processing DeleteTablet for tablet f473f3ef32cf48b19469f277f12833d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:27:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55696
I20260926 23:27:29.278795 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 16480
I20260926 23:27:29.279919 16932 tablet_replica.cc:330] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: stopping tablet replica
I20260926 23:27:29.280444 16932 raft_consensus.cc:2229] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:27:29.281181 16932 raft_consensus.cc:2258] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:27:29.282305 16934 tablet_replica.cc:330] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: stopping tablet replica
I20260926 23:27:29.282739 16934 raft_consensus.cc:2229] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:27:29.283391 16934 raft_consensus.cc:2258] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:27:29.283721 16932 ts_tablet_manager.cc:1905] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:27:29.285135 16934 ts_tablet_manager.cc:1905] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:27:29.291347 16932 ts_tablet_manager.cc:1918] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:27:29.291601 16932 log.cc:1196] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/f473f3ef32cf48b19469f277f12833d2
I20260926 23:27:29.292261 16932 ts_tablet_manager.cc:1939] T f473f3ef32cf48b19469f277f12833d2 P 17ae87cdee2e476e8e84134920028488: Deleting consensus metadata
I20260926 23:27:29.293473 16934 ts_tablet_manager.cc:1918] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:27:29.293730 16934 log.cc:1196] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/f473f3ef32cf48b19469f277f12833d2
I20260926 23:27:29.294271 16427 catalog_manager.cc:4841] TS 17ae87cdee2e476e8e84134920028488 (127.0.97.1:41129): tablet f473f3ef32cf48b19469f277f12833d2 (table test-table [id=a4902d30994a4e34a223a797c32c605f]) successfully deleted
I20260926 23:27:29.294369 16934 ts_tablet_manager.cc:1939] T f473f3ef32cf48b19469f277f12833d2 P 2d88fef63af4453cbbd4189b48942199: Deleting consensus metadata
I20260926 23:27:29.296001 16933 tablet_replica.cc:330] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: stopping tablet replica
I20260926 23:27:29.296017 16428 catalog_manager.cc:4841] TS 2d88fef63af4453cbbd4189b48942199 (127.0.97.2:38729): tablet f473f3ef32cf48b19469f277f12833d2 (table test-table [id=a4902d30994a4e34a223a797c32c605f]) successfully deleted
I20260926 23:27:29.297392 16933 raft_consensus.cc:2229] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:27:29.297482 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.279724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.0.97.1:41129)
0926 23:27:29.279904 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.279911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.280146 (+ 235us) server_negotiation.cc:407] Connection header received
0926 23:27:29.280346 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.280350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.280435 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.280557 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:29.282567 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.283688 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.285725 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.286388 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.286532 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.289873 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:29.289885 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:29.289901 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:29.289944 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:29.293930 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.295264 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.295276 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.295282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.295440 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.295805 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.295810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.295814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.296311 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:29.296617 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.296911 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.297114 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:27:29.297860 16933 raft_consensus.cc:2258] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:27:29.298177 16624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:29.300071 16933 ts_tablet_manager.cc:1905] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:27:29.307255 16933 ts_tablet_manager.cc:1918] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:27:29.307421 16933 log.cc:1196] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/f473f3ef32cf48b19469f277f12833d2
I20260926 23:27:29.308002 16933 ts_tablet_manager.cc:1939] T f473f3ef32cf48b19469f277f12833d2 P 2118d46059674c61acdbf973137de199: Deleting consensus metadata
I20260926 23:27:29.309957 16430 catalog_manager.cc:4841] TS 2118d46059674c61acdbf973137de199 (127.0.97.3:35609): tablet f473f3ef32cf48b19469f277f12833d2 (table test-table [id=a4902d30994a4e34a223a797c32c605f]) successfully deleted
W20260926 23:27:29.310462 16430 catalog_manager.cc:4808] TS 2118d46059674c61acdbf973137de199 (127.0.97.3:35609): delete failed for tablet f473f3ef32cf48b19469f277f12833d2 because the tablet was not found. No further retry: Not found: Tablet not found: f473f3ef32cf48b19469f277f12833d2
I20260926 23:27:29.389262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 16480
I20260926 23:27:29.413451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 16633
I20260926 23:27:29.427675 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.414210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.97.2:38729)
0926 23:27:29.414408 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.414420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.414440 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:27:29.414660 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.414665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.414726 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.414837 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:29.416079 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.417049 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.418383 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.418888 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.419017 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.421735 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:29.421751 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:29.421765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:29.421802 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:29.424987 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.425622 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.425631 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.425634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.425786 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.426151 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.426156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.426159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.426508 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:29.426845 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.427098 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.427335 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:27:29.428362 16756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:29.538915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 16633
I20260926 23:27:29.558542 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 16764
I20260926 23:27:29.572034 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:29.559325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.97.3:35609)
0926 23:27:29.559500 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 23:27:29.559507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:27:29.559528 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:27:29.559716 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:27:29.559726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:27:29.559795 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:27:29.559894 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:27:29.561259 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.561992 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.563424 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:27:29.563819 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:27:29.563945 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:27:29.566625 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:27:29.566635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:27:29.566650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:27:29.566683 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:27:29.569385 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.570058 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.570065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.570067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.570190 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:27:29.570561 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:27:29.570567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:27:29.570570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:27:29.570903 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:27:29.571182 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 23:27:29.571489 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 23:27:29.571705 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:27:29.572620 16887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:29.661903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 16764
I20260926 23:27:29.680441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 16410
I20260926 23:27:29.681624 16473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:27:29.795855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 16410
I20260926 23:27:49.816980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--webserver_interface=127.0.97.62
--webserver_port=38901
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:50.145881 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:50.146221 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:50.146396 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:50.153594 16939 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:27:50.153723 16939 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:27:50.153815 16939 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:27:50.153920 16939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:27:50.154026 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:50.154145 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:50.154227 16939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:27:50.154304 16939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:27:50.164219 16939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:50.165611 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:50.166989 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:50.179502 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:50.179575 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:50.179553 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:50.180186 16939 server_base.cc:1034] running on GCE node
I20260926 23:27:50.181092 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:50.182467 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:50.183748 16939 hybrid_clock.cc:648] HybridClock initialized: now 1790465270183678 us; error 38 us; skew 500 ppm
I20260926 23:27:50.184124 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:50 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:50.187352 16939 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:27:50.188838 16939 webserver.cc:460] Webserver started at http://127.0.97.62:38901/ using document root <none> and password file <none>
I20260926 23:27:50.189509 16939 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:50.189623 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:50.195621 16939 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:27:50.198596 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:50.199971 16939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:27:50.200165 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "158c634d308041ac8440a3c53985b275"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "f6c0b3bf360f750f2d50e7538cc2d393"
server_key_iv: "a4ccfb4edad1366e3e4b13f3562a3f39"
server_key_version: "encryptionkey@0"
I20260926 23:27:50.201218 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:50.216329 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:50.220837 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:50.221169 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:50.241925 16939 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45365
I20260926 23:27:50.241954 17005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45365 every 8 connection(s)
I20260926 23:27:50.243996 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:27:50.250756 17006 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:27:50.251219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 16939
I20260926 23:27:50.252147 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41129
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45365
--webserver_port=36201
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:27:50.254091 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Bootstrap starting.
I20260926 23:27:50.274075 17006 log.cc:824] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Log is configured to *not* fsync() on all Append() calls
I20260926 23:27:50.289659 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:27:50.290184 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Bootstrap complete.
I20260926 23:27:50.299105 17006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:50.300014 17006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158c634d308041ac8440a3c53985b275, State: Initialized, Role: FOLLOWER
I20260926 23:27:50.300668 17006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:50.300989 17006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:27:50.301112 17006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:27:50.301301 17006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:27:50.305400 17006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:50.305893 17006 leader_election.cc:302] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 158c634d308041ac8440a3c53985b275; no voters:
I20260926 23:27:50.306639 17006 leader_election.cc:288] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 23:27:50.306770 17010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 23:27:50.308023 17010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [term 3 LEADER]: Becoming Leader. State: Replica: 158c634d308041ac8440a3c53985b275, State: Running, Role: LEADER
I20260926 23:27:50.308660 17010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260926 23:27:50.309090 17006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:27:50.311193 17012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158c634d308041ac8440a3c53985b275. Latest consensus state: current_term: 3 leader_uuid: "158c634d308041ac8440a3c53985b275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260926 23:27:50.311599 17012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:27:50.312007 17011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "158c634d308041ac8440a3c53985b275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158c634d308041ac8440a3c53985b275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260926 23:27:50.312386 17011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:27:50.335127 17017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:27:50.345229 17017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:27:50.346871 17017 catalog_manager.cc:1259] Loaded cluster ID: 8a6c35a3732b424f916f0f183787b5da
I20260926 23:27:50.347052 17017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:27:50.351485 17017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:27:50.353278 17017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Loaded TSK: 0
I20260926 23:27:50.353549 17017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 158c634d308041ac8440a3c53985b275: Loaded TSK: 1
F20260926 23:27:50.354935 17017 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f18713cc9ad google::LogMessage::Fail() at ??:0
@ 0x7f18713cf88a google::LogMessage::SendToLog() at ??:0
@ 0x7f18713cc4de google::LogMessage::Flush() at ??:0
@ 0x7f18713cda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18751307d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f187511f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1886dfb1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1886dfa783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1886e862a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1886e860c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1886ed05e9 std::function<>::operator()() at ??:0
@ 0x7f1886dfecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1886dfcc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1886e84610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1886e84439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1874018345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f187401b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f187401b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1873ff6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f18772d06db start_thread at ??:0
@ 0x7f187026661f clone at ??:0
W20260926 23:27:50.645539 17008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:50.645845 17008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:50.646020 17008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:50.653348 17008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:50.653499 17008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:50.653671 17008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:27:50.663371 17008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36201
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:50.664669 17008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:50.666047 17008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:50.679134 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:50.679080 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:50.679098 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:50.680169 17008 server_base.cc:1034] running on GCE node
I20260926 23:27:50.680835 17008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:50.681737 17008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:50.682957 17008 hybrid_clock.cc:648] HybridClock initialized: now 1790465270682920 us; error 50 us; skew 500 ppm
I20260926 23:27:50.683259 17008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:50 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:50.686585 17008 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:27:50.687973 17008 webserver.cc:460] Webserver started at http://127.0.97.1:36201/ using document root <none> and password file <none>
I20260926 23:27:50.688659 17008 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:50.688861 17008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:50.694201 17008 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 23:27:50.696995 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:50.698354 17008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:27:50.698532 17008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "17ae87cdee2e476e8e84134920028488"
format_stamp: "Formatted at 2026-09-26 23:27:04 on dist-test-slave-q5g8"
server_key: "b63adbd5eeec82028266310df4686fb6"
server_key_iv: "67309d01533ad25f486c74f2747d40ae"
server_key_version: "encryptionkey@0"
I20260926 23:27:50.699530 17008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:50.715248 17008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:50.719128 17008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:50.719394 17008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:50.720324 17008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:50.721725 17008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:50.721845 17008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:50.721982 17008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:50.722090 17008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:50.756714 17008 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41129
I20260926 23:27:50.756771 17175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41129 every 8 connection(s)
I20260926 23:27:50.758143 17008 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:27:50.762969 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:50.761105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:39453)
0926 23:27:50.761898 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:50.762117 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":305,"threads_started":1}
W20260926 23:27:50.763612 17176 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45365 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45365: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
I20260926 23:27:50.764400 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:50.764100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:58081)
0926 23:27:50.764226 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:50.764288 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:27:50.765164 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:50.764882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:38649)
0926 23:27:50.764998 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:50.765065 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 23:27:50.765404 17176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:27:50.766077 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 17008
I20260926 23:27:50.766742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38729
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45365
--webserver_port=40285
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:51.086700 17179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:51.086943 17179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:51.087034 17179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:51.094019 17179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:51.094116 17179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:51.094223 17179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:27:51.103559 17179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40285
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:51.104799 17179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:51.106006 17179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:51.118917 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:51.119019 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:51.119027 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:51.119655 17179 server_base.cc:1034] running on GCE node
I20260926 23:27:51.120234 17179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:51.121225 17179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:51.122467 17179 hybrid_clock.cc:648] HybridClock initialized: now 1790465271122414 us; error 52 us; skew 500 ppm
I20260926 23:27:51.122782 17179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:51 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:51.125902 17179 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:27:51.127350 17179 webserver.cc:460] Webserver started at http://127.0.97.2:40285/ using document root <none> and password file <none>
I20260926 23:27:51.128012 17179 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:51.128157 17179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:51.133416 17179 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:27:51.136132 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:51.137465 17179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:27:51.137632 17179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "2d88fef63af4453cbbd4189b48942199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "d8d69629e43026f3ee20a9ae94d7d10d"
server_key_iv: "fb091a5f18a8ebf059f948841c876201"
server_key_version: "encryptionkey@0"
I20260926 23:27:51.138600 17179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:51.161496 17179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:51.165745 17179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:51.166018 17179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:51.167138 17179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:51.168861 17179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:51.168974 17179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:51.169081 17179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:51.169178 17179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:51.211206 17179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38729
I20260926 23:27:51.211258 17306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38729 every 8 connection(s)
I20260926 23:27:51.212671 17179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:27:51.217713 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:51.215848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:43775)
0926 23:27:51.216576 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:51.216871 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":288,"threads_started":1}
W20260926 23:27:51.218330 17307 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45365 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45365: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
I20260926 23:27:51.219046 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:51.218734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:43791)
0926 23:27:51.218887 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:51.218949 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:27:51.219515 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 17179
I20260926 23:27:51.219880 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:51.219559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:40483)
0926 23:27:51.219699 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:51.219776 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:27:51.220098 17307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:27:51.220197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35609
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45365
--webserver_port=37337
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:27:51.543233 17310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:27:51.543501 17310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:27:51.543592 17310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:27:51.550837 17310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:27:51.550930 17310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:27:51.551067 17310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:27:51.560439 17310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37337
--tserver_master_addrs=127.0.97.62:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:27:51.561715 17310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:27:51.562927 17310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:27:51.575722 17315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:51.575778 17318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:27:51.575812 17316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:27:51.576329 17310 server_base.cc:1034] running on GCE node
I20260926 23:27:51.576943 17310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:27:51.577750 17310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:27:51.578961 17310 hybrid_clock.cc:648] HybridClock initialized: now 1790465271578912 us; error 42 us; skew 500 ppm
I20260926 23:27:51.579272 17310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:27:51 dist-test-slave-q5g8 krb5kdc[15857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:27:51.582490 17310 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:27:51.583864 17310 webserver.cc:460] Webserver started at http://127.0.97.3:37337/ using document root <none> and password file <none>
I20260926 23:27:51.584491 17310 fs_manager.cc:359] Metadata directory not provided
I20260926 23:27:51.584587 17310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:27:51.589761 17310 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:27:51.592392 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:51.593734 17310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:27:51.593922 17310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "2118d46059674c61acdbf973137de199"
format_stamp: "Formatted at 2026-09-26 23:27:05 on dist-test-slave-q5g8"
server_key: "4691e023bf352c21b0d91d6ae8ba8a2b"
server_key_iv: "647ff2c25d135e4d9b3372df3b0d4e24"
server_key_version: "encryptionkey@0"
I20260926 23:27:51.594879 17310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:27:51.619380 17310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:27:51.623711 17310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:27:51.624004 17310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:27:51.624979 17310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:27:51.626276 17310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:27:51.626403 17310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:51.626537 17310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:27:51.626626 17310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:27:51.660247 17310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35609
I20260926 23:27:51.660337 17437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35609 every 8 connection(s)
I20260926 23:27:51.661753 17310 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:27:51.663053 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 17310
I20260926 23:27:51.666874 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:51.665092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:34753)
0926 23:27:51.665796 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:51.666011 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":285,"threads_started":1}
W20260926 23:27:51.667514 17438 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45365 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45365: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
I20260926 23:27:51.668308 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:51.668010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:42089)
0926 23:27:51.668142 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:51.668206 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 23:27:51.669109 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:51.668835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:60837)
0926 23:27:51.668941 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:51.669009 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 23:27:51.669384 17438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:27:51.766904 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:51.766111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:58963)
0926 23:27:51.766581 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:51.766667 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260926 23:27:52.221629 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:52.220772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:43099)
0926 23:27:52.221237 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:52.221376 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260926 23:27:52.670732 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:52.669900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:44043)
0926 23:27:52.670390 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:52.670482 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":218,"threads_started":1}
I20260926 23:27:52.768556 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:52.767827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:46681)
0926 23:27:52.768290 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:52.768377 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
I20260926 23:27:53.223237 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:53.222552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:46815)
0926 23:27:53.222995 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:53.223074 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":222,"threads_started":1}
I20260926 23:27:53.672462 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:53.671722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:33713)
0926 23:27:53.672171 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:53.672250 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":221,"threads_started":1}
I20260926 23:27:53.770191 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:53.769456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:39449)
0926 23:27:53.769940 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:53.770021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":252,"threads_started":1}
I20260926 23:27:54.224974 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:54.224099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:59823)
0926 23:27:54.224673 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:54.224799 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":364,"threads_started":1}
I20260926 23:27:54.674129 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:54.673416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:46289)
0926 23:27:54.673880 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:54.673961 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":244,"threads_started":1}
I20260926 23:27:54.772074 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:54.771192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:40587)
0926 23:27:54.771759 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:54.771839 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":324,"threads_started":1}
I20260926 23:27:55.226661 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:55.225886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:39339)
0926 23:27:55.226410 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:55.226491 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":237,"threads_started":1}
I20260926 23:27:55.675980 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:55.675135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:57113)
0926 23:27:55.675688 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:55.675774 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":325,"threads_started":1}
I20260926 23:27:55.773977 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:55.773087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:37009)
0926 23:27:55.773725 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:55.773805 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":388,"threads_started":1}
I20260926 23:27:56.228457 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:56.227566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:58719)
0926 23:27:56.228119 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:56.228229 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":257,"threads_started":1}
I20260926 23:27:56.677718 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:56.677005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:48611)
0926 23:27:56.677477 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:56.677557 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
I20260926 23:27:56.775503 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:56.774809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:39437)
0926 23:27:56.775235 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:56.775318 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260926 23:27:57.230261 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:57.229434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:56727)
0926 23:27:57.229945 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:57.230079 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260926 23:27:57.679435 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:57.678691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:48517)
0926 23:27:57.679131 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:57.679251 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260926 23:27:57.777320 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:57.776471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:48649)
0926 23:27:57.777018 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:57.777097 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":280,"threads_started":1}
I20260926 23:27:58.232525 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:58.231429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:44477)
0926 23:27:58.232095 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:58.232198 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":339,"threads_started":1}
I20260926 23:27:58.681399 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:58.680516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:45505)
0926 23:27:58.681085 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:58.681168 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":289,"threads_started":1}
I20260926 23:27:58.779173 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:58.778424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:53241)
0926 23:27:58.778915 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:58.779003 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260926 23:27:59.234499 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:59.233725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:53473)
0926 23:27:59.234221 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:59.234314 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260926 23:27:59.683503 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:59.682597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:60037)
0926 23:27:59.683182 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:59.683303 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":293,"threads_started":1}
I20260926 23:27:59.780995 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:27:59.780161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:58849)
0926 23:27:59.780667 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 23:27:59.780784 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
I20260926 23:28:00.236617 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:00.235528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:47709)
0926 23:28:00.236219 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:00.236350 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":310,"threads_started":1}
I20260926 23:28:00.685578 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:00.684633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:34489)
0926 23:28:00.685284 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:00.685392 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":354,"threads_started":1}
I20260926 23:28:00.783143 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:00.782149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:37133)
0926 23:28:00.782793 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:00.782913 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":374,"threads_started":1}
I20260926 23:28:01.238794 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:01.237911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:51185)
0926 23:28:01.238457 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:01.238550 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":319,"threads_started":1}
I20260926 23:28:01.687481 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:01.686652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:43147)
0926 23:28:01.687203 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:01.687301 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":319,"threads_started":1}
I20260926 23:28:01.785261 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:01.784363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:38765)
0926 23:28:01.784970 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:01.785073 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":259,"threads_started":1}
I20260926 23:28:02.240981 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:02.240033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:43665)
0926 23:28:02.240595 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:02.240751 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":283,"threads_started":1}
I20260926 23:28:02.689800 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:02.688816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:50019)
0926 23:28:02.689411 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:02.689533 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":303,"threads_started":1}
I20260926 23:28:02.787175 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:02.786430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:34763)
0926 23:28:02.786925 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:02.787013 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260926 23:28:03.243108 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:03.242273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:40093)
0926 23:28:03.242813 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:03.242930 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
I20260926 23:28:03.691709 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:03.690930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:56603)
0926 23:28:03.691440 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:03.691544 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":280,"threads_started":1}
I20260926 23:28:03.789187 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:03.788296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:45321)
0926 23:28:03.788877 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:03.788976 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":237,"threads_started":1}
I20260926 23:28:04.245160 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:04.244147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:38677)
0926 23:28:04.244828 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:04.244940 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260926 23:28:04.693811 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:04.692978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:38743)
0926 23:28:04.693517 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:04.693614 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":249,"threads_started":1}
I20260926 23:28:04.790964 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:04.790235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:54323)
0926 23:28:04.790695 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:04.790776 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260926 23:28:05.247048 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:05.246239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:51687)
0926 23:28:05.246765 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:05.246859 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260926 23:28:05.695593 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:05.694805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:60055)
0926 23:28:05.695304 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:05.695398 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260926 23:28:05.793241 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:05.792353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:47161)
0926 23:28:05.792956 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:05.793052 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":343,"threads_started":1}
I20260926 23:28:06.249011 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:06.248168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:59419)
0926 23:28:06.248639 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:06.248818 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":250,"threads_started":1}
I20260926 23:28:06.674355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 17008
I20260926 23:28:06.697817 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:06.697024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:39435)
0926 23:28:06.697549 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:06.697631 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":286,"threads_started":1}
I20260926 23:28:06.701750 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:06.675631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.0.97.1:41129)
0926 23:28:06.676292 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:06.676311 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:06.676364 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:28:06.678050 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:06.678060 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:06.678494 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:06.678873 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:06.680498 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:06.681837 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:06.683156 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:06.683871 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:06.684085 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:06.694492 (+ 10407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:06.694501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:06.694521 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:06.694570 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:06.698169 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:06.698971 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:06.698986 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:06.698994 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:06.699208 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:06.699630 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:06.699635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:06.699638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:06.700213 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:06.700556 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:06.701026 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 23:28:06.701472 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":305,"threads_started":1}
I20260926 23:28:06.703387 17171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:06.789357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 17008
I20260926 23:28:06.808377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 17179
I20260926 23:28:06.827469 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:06.809710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.0.97.2:38729)
0926 23:28:06.810420 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:06.810435 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:06.810477 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:28:06.812154 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:06.812164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:06.812610 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:06.813107 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:06.814553 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:06.815950 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:06.817517 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:06.818248 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:06.818400 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:06.821141 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:06.821149 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:06.821168 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:06.821209 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:06.824190 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:06.825064 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:06.825071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:06.825079 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:06.825257 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:06.825650 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:06.825654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:06.825656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:06.826203 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:06.826433 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:06.826845 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 23:28:06.827184 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":339,"threads_started":1}
I20260926 23:28:06.829037 17302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:06.913533 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 17179
I20260926 23:28:06.931131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 17310
I20260926 23:28:06.949035 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:06.932124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.0.97.3:35609)
0926 23:28:06.932831 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:06.932846 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:06.932889 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:28:06.934536 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:06.934544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:06.934986 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:06.935277 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:06.936545 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:06.937854 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:06.939129 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:06.939794 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:06.939926 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:06.942656 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:06.942664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:06.942679 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:06.942713 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:06.945551 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:06.946280 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:06.946286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:06.946292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:06.946453 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:06.946957 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:06.946962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:06.946964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:06.947545 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:06.947779 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:06.948237 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 23:28:06.948573 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":360,"threads_started":1}
I20260926 23:28:06.950536 17433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:07.036268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 17310
2026-09-26T23:28:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64875 ms)
[----------] 30 tests from SecurityITest (308150 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:07 dist-test-slave-q5g8 krb5kdc[17495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:07 dist-test-slave-q5g8 krb5kdc[17495](info): set up 2 sockets
Sep 26 23:28:07 dist-test-slave-q5g8 krb5kdc[17495](info): commencing operation
krb5kdc: starting...
W20260926 23:28:08.136767 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:08 dist-test-slave-q5g8 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:28:08.388357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45947
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:08.716078 17508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:08.716357 17508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:08.716529 17508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:08.723735 17508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:08.723835 17508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:08.723929 17508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:08.724032 17508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:08.724123 17508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:08.733754 17508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:08.735005 17508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:08.736304 17508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:08.748927 17513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:08.748999 17514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:08.748970 17516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:08.749454 17508 server_base.cc:1034] running on GCE node
I20260926 23:28:08.750486 17508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:08.751884 17508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:08.753078 17508 hybrid_clock.cc:648] HybridClock initialized: now 1790465288753028 us; error 42 us; skew 500 ppm
I20260926 23:28:08.753439 17508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:08 dist-test-slave-q5g8 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:08.756464 17508 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:28:08.758013 17508 webserver.cc:460] Webserver started at http://127.0.97.62:39341/ using document root <none> and password file <none>
I20260926 23:28:08.758719 17508 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:08.758865 17508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:08.759171 17508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:08.762256 17508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "4d404a68b2e94e4985ceac4da1e8ac87"
format_stamp: "Formatted at 2026-09-26 23:28:08 on dist-test-slave-q5g8"
server_key: "3850ad79be416b38c2e6594031a3e41b"
server_key_iv: "f9194df5b6e4eedc5051a6203d9071bc"
server_key_version: "encryptionkey@0"
I20260926 23:28:08.762990 17508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "4d404a68b2e94e4985ceac4da1e8ac87"
format_stamp: "Formatted at 2026-09-26 23:28:08 on dist-test-slave-q5g8"
server_key: "3850ad79be416b38c2e6594031a3e41b"
server_key_iv: "f9194df5b6e4eedc5051a6203d9071bc"
server_key_version: "encryptionkey@0"
I20260926 23:28:08.768025 17508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 23:28:08.771420 17523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:08.772642 17508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 23:28:08.772881 17508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "4d404a68b2e94e4985ceac4da1e8ac87"
format_stamp: "Formatted at 2026-09-26 23:28:08 on dist-test-slave-q5g8"
server_key: "3850ad79be416b38c2e6594031a3e41b"
server_key_iv: "f9194df5b6e4eedc5051a6203d9071bc"
server_key_version: "encryptionkey@0"
I20260926 23:28:08.773078 17508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:08.784732 17508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:08.789429 17508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:08.789742 17508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:08.807487 17508 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45947
I20260926 23:28:08.807505 17574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45947 every 8 connection(s)
I20260926 23:28:08.809055 17508 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:08.811893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 17508
I20260926 23:28:08.812107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:08.812812 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:08.812871 388 external_mini_cluster.cc:1428] Setting key 127a8753946b4112e8cc736a1b89ce31
I20260926 23:28:08.826774 17575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87: Bootstrap starting.
Sep 26 23:28:08 dist-test-slave-q5g8 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:08.829986 17575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:08.831024 17575 log.cc:824] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:08.834084 17575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87: No bootstrap required, opened a new log
I20260926 23:28:08.837589 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:08.814905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.0.97.62:45947)
0926 23:28:08.815792 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:08.815807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:08.815855 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:28:08.817751 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:08.817777 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:08.818257 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:08.818734 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:08.820225 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:08.822181 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:08.823632 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:08.824768 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:08.824960 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:08.829184 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:08.829194 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:08.829210 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:08.829275 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:08.833232 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:08.833989 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:08.833998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:08.834006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:08.834237 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:08.834656 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:08.834664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:08.834683 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:08.835233 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:08.835561 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:08.836068 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 23:28:08.836518 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":372,"threads_started":1}
I20260926 23:28:08.842425 17575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260926 23:28:08.842866 17575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:08.842973 17575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d404a68b2e94e4985ceac4da1e8ac87, State: Initialized, Role: FOLLOWER
I20260926 23:28:08.843487 17575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260926 23:28:08.843721 17575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:08.843899 17575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:08.844067 17575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:08.846460 17575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260926 23:28:08.846922 17575 leader_election.cc:302] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d404a68b2e94e4985ceac4da1e8ac87; no voters:
I20260926 23:28:08.847496 17575 leader_election.cc:288] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:08.847672 17580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:08.848466 17580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d404a68b2e94e4985ceac4da1e8ac87, State: Running, Role: LEADER
I20260926 23:28:08.849031 17580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } }
I20260926 23:28:08.849634 17575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:08.851878 17582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d404a68b2e94e4985ceac4da1e8ac87. Latest consensus state: current_term: 1 leader_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } } }
I20260926 23:28:08.851837 17581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d404a68b2e94e4985ceac4da1e8ac87" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45947 } } }
I20260926 23:28:08.852393 17582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:08.852506 17581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87 [sys.catalog]: This master's current role is: LEADER
W20260926 23:28:08.862321 17596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:28:08.862470 17596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:28:08.862651 17586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:08.864347 17586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:08.872632 17586 catalog_manager.cc:1347] Generated new cluster ID: f6cf74934b7a43ebb904dd494c508a3b
I20260926 23:28:08.872797 17586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:08.882534 17586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:08.883845 17586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:08.893203 17586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87: Generated new TSK 0
I20260926 23:28:08.894042 17586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:28:08.957820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:09.290769 17601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:09.291033 17601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:09.291123 17601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:09.298348 17601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:09.298446 17601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:09.298557 17601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:28:09.307878 17601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:09.309139 17601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:09.310489 17601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:09.324795 17606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:09.324754 17607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:09.324834 17609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:09.325491 17601 server_base.cc:1034] running on GCE node
I20260926 23:28:09.326084 17601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:09.327008 17601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:09.328276 17601 hybrid_clock.cc:648] HybridClock initialized: now 1790465289328223 us; error 62 us; skew 500 ppm
I20260926 23:28:09.328593 17601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:09 dist-test-slave-q5g8 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:09.335837 17601 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:28:09.337808 17601 webserver.cc:460] Webserver started at http://127.0.97.1:42141/ using document root <none> and password file <none>
I20260926 23:28:09.338608 17601 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:09.338755 17601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:09.339072 17601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:09.342038 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "7e77b42b2043467bb4480d027e1f07e9"
format_stamp: "Formatted at 2026-09-26 23:28:09 on dist-test-slave-q5g8"
server_key: "4ca42ea5ee0fe00edcc2411c886086c1"
server_key_iv: "87bb8727b8f9bb88694c0eaf0ef6b6fe"
server_key_version: "encryptionkey@0"
I20260926 23:28:09.342818 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7e77b42b2043467bb4480d027e1f07e9"
format_stamp: "Formatted at 2026-09-26 23:28:09 on dist-test-slave-q5g8"
server_key: "4ca42ea5ee0fe00edcc2411c886086c1"
server_key_iv: "87bb8727b8f9bb88694c0eaf0ef6b6fe"
server_key_version: "encryptionkey@0"
I20260926 23:28:09.347862 17601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:28:09.351656 17616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:09.353145 17601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:28:09.353276 17601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "7e77b42b2043467bb4480d027e1f07e9"
format_stamp: "Formatted at 2026-09-26 23:28:09 on dist-test-slave-q5g8"
server_key: "4ca42ea5ee0fe00edcc2411c886086c1"
server_key_iv: "87bb8727b8f9bb88694c0eaf0ef6b6fe"
server_key_version: "encryptionkey@0"
I20260926 23:28:09.353435 17601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:09.369176 17601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:09.373885 17601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:09.374123 17601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:09.375049 17601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:09.376513 17601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:09.376585 17601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:09.376798 17601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:09.376854 17601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:09.410285 17601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34971
I20260926 23:28:09.410372 17728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34971 every 8 connection(s)
I20260926 23:28:09.411592 17601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:28:09.412425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 17601
I20260926 23:28:09.412751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:28:09.414578 388 external_mini_cluster.cc:1428] Setting key 668e048fc425ca24f6e86b36a24aaceb
Sep 26 23:28:09 dist-test-slave-q5g8 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:09.442375 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:09.417239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44941 (local address 127.0.97.62:45947)
0926 23:28:09.417787 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:09.417795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:09.417819 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:28:09.420652 (+ 2833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:09.420658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:09.420766 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:09.420894 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:09.423597 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.425117 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.428311 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.429190 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.429329 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:09.434237 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:09.434247 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:09.434254 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:09.434307 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:09.438165 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:09.439217 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:09.439225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:09.439231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:09.439378 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:09.439977 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:09.439983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:09.439987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:09.440327 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:09.440484 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:09.441476 (+ 992us) server_negotiation.cc:299] Negotiation successful
0926 23:28:09.441842 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":399,"threads_started":1}
I20260926 23:28:09.443287 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:09.415791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45947 (local address 127.0.97.1:44941)
0926 23:28:09.416933 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:09.417008 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:09.420103 (+ 3095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:09.421159 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:09.421174 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:09.422324 (+ 1150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:09.423376 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:09.423403 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.425343 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.425350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:09.428109 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:09.428127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.429649 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.429658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:09.429999 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:09.430814 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:09.430850 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:09.433947 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:09.438344 (+ 4397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:09.438355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:09.438380 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:09.439054 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:09.439511 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:09.439521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:09.439523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:09.439827 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:09.440473 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:09.440482 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:09.441003 (+ 521us) client_negotiation.cc:769] Sending connection context
0926 23:28:09.441327 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 23:28:09.441887 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":422,"threads_started":1}
I20260926 23:28:09.445077 17729 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45947
I20260926 23:28:09.445428 17729 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:09.446403 17729 heartbeater.cc:511] Master 127.0.97.62:45947 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:28:09.449229 17540 ts_manager.cc:194] Registered new tserver with Master: 7e77b42b2043467bb4480d027e1f07e9 (127.0.97.1:34971)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:28:09.451800 17540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:28:09.481017 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:09.803356 17735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:09.803671 17735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:09.803839 17735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:09.813494 17735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:09.813629 17735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:09.813812 17735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:28:09.829231 17735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:09.830709 17735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:09.832089 17735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:09.845505 17740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:09.845440 17741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:09.845476 17743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:09.846290 17735 server_base.cc:1034] running on GCE node
I20260926 23:28:09.846868 17735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:09.847886 17735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:09.849099 17735 hybrid_clock.cc:648] HybridClock initialized: now 1790465289849048 us; error 46 us; skew 500 ppm
I20260926 23:28:09.849411 17735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:09 dist-test-slave-q5g8 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:09.852732 17735 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:28:09.854163 17735 webserver.cc:460] Webserver started at http://127.0.97.2:38393/ using document root <none> and password file <none>
I20260926 23:28:09.854832 17735 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:09.854975 17735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:09.855314 17735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:09.858500 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "b5d655d433b14ad79950943f91e2da77"
format_stamp: "Formatted at 2026-09-26 23:28:09 on dist-test-slave-q5g8"
server_key: "f7de0db278b7fc1ac2eed128238b8569"
server_key_iv: "4864828d74c6ae250dfa35b4b1c4f388"
server_key_version: "encryptionkey@0"
I20260926 23:28:09.859256 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b5d655d433b14ad79950943f91e2da77"
format_stamp: "Formatted at 2026-09-26 23:28:09 on dist-test-slave-q5g8"
server_key: "f7de0db278b7fc1ac2eed128238b8569"
server_key_iv: "4864828d74c6ae250dfa35b4b1c4f388"
server_key_version: "encryptionkey@0"
I20260926 23:28:09.864234 17735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:28:09.867525 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:09.868816 17735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:28:09.868970 17735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "b5d655d433b14ad79950943f91e2da77"
format_stamp: "Formatted at 2026-09-26 23:28:09 on dist-test-slave-q5g8"
server_key: "f7de0db278b7fc1ac2eed128238b8569"
server_key_iv: "4864828d74c6ae250dfa35b4b1c4f388"
server_key_version: "encryptionkey@0"
I20260926 23:28:09.869163 17735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:09.888934 17735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:09.893148 17735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:09.893517 17735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:09.894449 17735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:09.895706 17735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:09.895819 17735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:09.895972 17735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:09.896045 17735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:09.930174 17735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44351
I20260926 23:28:09.930276 17862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44351 every 8 connection(s)
I20260926 23:28:09.931439 17735 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:28:09.934381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 17735
I20260926 23:28:09.934556 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:28:09.935247 388 external_mini_cluster.cc:1428] Setting key ddf42798529dd630e8c4fb0209a1af43
Sep 26 23:28:09 dist-test-slave-q5g8 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:09.955449 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:09.933900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44505 (local address 127.0.97.62:45947)
0926 23:28:09.934125 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:09.934132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:09.935507 (+ 1375us) server_negotiation.cc:407] Connection header received
0926 23:28:09.937731 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:09.937736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:09.937809 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:09.937891 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:09.940021 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.940815 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.943229 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.943638 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.943726 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:09.948252 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:09.948261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:09.948270 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:09.948317 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:09.952033 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:09.952824 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:09.952832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:09.952838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:09.952983 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:09.953409 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:09.953417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:09.953423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:09.953822 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:09.954000 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:09.954713 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 23:28:09.954996 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:28:09.956280 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:09.934443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45947 (local address 127.0.97.2:44505)
0926 23:28:09.935326 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:09.935383 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:09.937435 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:09.938185 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:09.938197 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:09.939019 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:09.939844 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:09.939870 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.940962 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.940971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:09.943084 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:09.943102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:09.943861 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:09.943870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:09.944193 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:09.945001 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:09.945043 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:09.948010 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:09.952179 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:09.952186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:09.952199 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:09.952628 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:09.953091 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:09.953094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:09.953097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:09.953288 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:09.953950 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:09.953956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:09.954285 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 23:28:09.954581 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 23:28:09.955069 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":343,"threads_started":1}
I20260926 23:28:09.957847 17863 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45947
I20260926 23:28:09.958173 17863 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:09.958915 17863 heartbeater.cc:511] Master 127.0.97.62:45947 requested a full tablet report, sending...
I20260926 23:28:09.960521 17540 ts_manager.cc:194] Registered new tserver with Master: b5d655d433b14ad79950943f91e2da77 (127.0.97.2:44351)
I20260926 23:28:09.961380 17540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44505
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:28:09.996469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:10.319635 17868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:10.320070 17868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:10.320227 17868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:10.331559 17868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:10.331707 17868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:10.331923 17868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:28:10.342092 17868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:10.343395 17868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:10.344668 17868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:10.357817 17873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:10.357874 17876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:10.358021 17874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:10.359063 17868 server_base.cc:1034] running on GCE node
I20260926 23:28:10.359589 17868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:10.360500 17868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:10.361719 17868 hybrid_clock.cc:648] HybridClock initialized: now 1790465290361677 us; error 38 us; skew 500 ppm
I20260926 23:28:10.362041 17868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:10 dist-test-slave-q5g8 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:10.365070 17868 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:28:10.366796 17868 webserver.cc:460] Webserver started at http://127.0.97.3:36659/ using document root <none> and password file <none>
I20260926 23:28:10.367481 17868 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:10.367617 17868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:10.367923 17868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:10.370713 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "4ad1373ab900492e9811b14bdfc3c019"
format_stamp: "Formatted at 2026-09-26 23:28:10 on dist-test-slave-q5g8"
server_key: "dcc7ef6d20453dd3a1a5f7b833622887"
server_key_iv: "1a23c42b51f622a3cd43e400a0e58239"
server_key_version: "encryptionkey@0"
I20260926 23:28:10.371484 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4ad1373ab900492e9811b14bdfc3c019"
format_stamp: "Formatted at 2026-09-26 23:28:10 on dist-test-slave-q5g8"
server_key: "dcc7ef6d20453dd3a1a5f7b833622887"
server_key_iv: "1a23c42b51f622a3cd43e400a0e58239"
server_key_version: "encryptionkey@0"
I20260926 23:28:10.376608 17868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:28:10.380064 17883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:10.381487 17868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:28:10.381623 17868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "4ad1373ab900492e9811b14bdfc3c019"
format_stamp: "Formatted at 2026-09-26 23:28:10 on dist-test-slave-q5g8"
server_key: "dcc7ef6d20453dd3a1a5f7b833622887"
server_key_iv: "1a23c42b51f622a3cd43e400a0e58239"
server_key_version: "encryptionkey@0"
I20260926 23:28:10.381788 17868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:10.400877 17868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:10.405164 17868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:10.405414 17868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:10.406332 17868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:10.407542 17868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:10.407612 17868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:10.407696 17868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:10.407742 17868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:10.440105 17868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33331
I20260926 23:28:10.440253 17995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33331 every 8 connection(s)
I20260926 23:28:10.441428 17868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:28:10.446926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 17868
I20260926 23:28:10.447113 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:28:10.447530 388 external_mini_cluster.cc:1428] Setting key f6edc5470a6f17f98b8fdd92194802ad
Sep 26 23:28:10 dist-test-slave-q5g8 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:10.456054 17729 heartbeater.cc:503] Master 127.0.97.62:45947 was elected leader, sending a full tablet report...
I20260926 23:28:10.461275 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.443972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51485 (local address 127.0.97.62:45947)
0926 23:28:10.444155 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.444160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.445360 (+ 1200us) server_negotiation.cc:407] Connection header received
0926 23:28:10.447143 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.447149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.447243 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.447356 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:10.448936 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.449852 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.451548 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.451951 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.452082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.455465 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:10.455475 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:10.455484 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:10.455516 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:10.458554 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.459218 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.459222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.459224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.459351 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.459709 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.459711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.459713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.459966 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:10.460077 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.460671 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.460942 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:28:10.462481 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.444439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45947 (local address 127.0.97.3:51485)
0926 23:28:10.445197 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.445237 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.446874 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.447589 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.447598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.448151 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:10.448761 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.448786 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.450002 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.450007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.451396 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.451408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.452186 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.452193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.452445 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.453078 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:10.453104 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:10.455260 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:10.458672 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:10.458679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:10.458691 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:10.459105 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:10.459443 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:10.459446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:10.459448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:10.459614 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:10.460070 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:10.460076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:10.460363 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.460603 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.461127 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":355,"threads_started":1}
I20260926 23:28:10.463773 17996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45947
I20260926 23:28:10.464022 17996 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:10.464658 17996 heartbeater.cc:511] Master 127.0.97.62:45947 requested a full tablet report, sending...
I20260926 23:28:10.465978 17540 ts_manager.cc:194] Registered new tserver with Master: 4ad1373ab900492e9811b14bdfc3c019 (127.0.97.3:33331)
I20260926 23:28:10.466777 17540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51485
I20260926 23:28:10.474555 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:28:10.491233 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.478837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.0.97.62:45947)
0926 23:28:10.479028 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.479036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.479435 (+ 399us) server_negotiation.cc:407] Connection header received
0926 23:28:10.479635 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.479645 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.479704 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.479815 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:10.481291 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.482004 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.483326 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.483733 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.483876 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.486331 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:10.486336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:10.486339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:10.486369 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:10.488585 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.489205 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.489212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.489215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.489341 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.489796 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.489800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.489802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.490093 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:10.490364 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.490689 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.490920 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 23:28:10.495028 17540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:28:10.501672 17540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:28:10.539099 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.513149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33331 (local address 127.0.0.1:34208)
0926 23:28:10.527993 (+ 14844us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.528019 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.528185 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.529759 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.529771 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.529812 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:10.530326 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.530342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.533043 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.533052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.535861 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.535876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.538154 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.538160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.538341 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.538406 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.538531 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.538714 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14657,"thread_start_us":22994,"threads_started":1}
I20260926 23:28:10.539130 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.514497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.0.97.3:33331)
0926 23:28:10.527428 (+ 12931us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.527441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.528349 (+ 908us) server_negotiation.cc:407] Connection header received
0926 23:28:10.528484 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.528496 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.528884 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.529129 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:10.530586 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.532795 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.536049 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.537670 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.537794 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.538022 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.538603 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.538769 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12731,"thread_start_us":12572,"threads_started":1}
I20260926 23:28:10.543437 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.514358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.97.2:44351)
0926 23:28:10.535207 (+ 20849us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.535216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.536542 (+ 1326us) server_negotiation.cc:407] Connection header received
0926 23:28:10.536799 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.536808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.537032 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.537263 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:10.537741 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.538888 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.541450 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.542388 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.542537 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.542749 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.542954 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.543109 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20678,"thread_start_us":20518,"threads_started":1}
I20260926 23:28:10.543480 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.513994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44351 (local address 127.0.0.1:45604)
0926 23:28:10.536436 (+ 22442us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.536455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.536579 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.537263 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.537268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.537292 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:10.537593 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.537602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.539061 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.539067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.541298 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.541310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.542582 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.542588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.542741 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.542783 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.542912 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.543041 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22312,"mutex_wait_us":1,"thread_start_us":12121,"threads_started":1}
I20260926 23:28:10.543797 17931 tablet_service.cc:1467] Processing CreateTablet for tablet 779ddef499e5496ba913c85e164a1cb2 (DEFAULT_TABLE table=test-table [id=15994b40ae934c54b994130ef6a11a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:10.544602 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.513034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34971 (local address 127.0.0.1:51824)
0926 23:28:10.526509 (+ 13475us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.526557 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.526715 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.536635 (+ 9920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.536641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.536676 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:10.537167 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.537181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.539034 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.539040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.540944 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.540958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.543873 (+ 2915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.543879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.544035 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.544078 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.544156 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.544278 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13217,"thread_start_us":16389,"threads_started":1}
I20260926 23:28:10.545445 17931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779ddef499e5496ba913c85e164a1cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:10.545871 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.513168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.0.97.1:34971)
0926 23:28:10.535246 (+ 22078us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.535258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.535294 (+ 36us) server_negotiation.cc:407] Connection header received
0926 23:28:10.535426 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.535438 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.535766 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.536661 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:10.537394 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.538888 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.541150 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.543648 (+ 2498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.545102 (+ 1454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.545336 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.545407 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.545573 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21909,"thread_start_us":23158,"threads_started":1}
I20260926 23:28:10.547253 17798 tablet_service.cc:1467] Processing CreateTablet for tablet 779ddef499e5496ba913c85e164a1cb2 (DEFAULT_TABLE table=test-table [id=15994b40ae934c54b994130ef6a11a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:10.548322 17798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779ddef499e5496ba913c85e164a1cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:10.549538 17664 tablet_service.cc:1467] Processing CreateTablet for tablet 779ddef499e5496ba913c85e164a1cb2 (DEFAULT_TABLE table=test-table [id=15994b40ae934c54b994130ef6a11a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:10.550885 17664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779ddef499e5496ba913c85e164a1cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:10.559403 18014 tablet_bootstrap.cc:492] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Bootstrap starting.
I20260926 23:28:10.561405 18015 tablet_bootstrap.cc:492] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Bootstrap starting.
I20260926 23:28:10.563127 18014 tablet_bootstrap.cc:654] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:10.564371 18014 log.cc:824] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:10.564756 18015 tablet_bootstrap.cc:654] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:10.566182 18015 log.cc:824] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:10.566291 18017 tablet_bootstrap.cc:492] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Bootstrap starting.
I20260926 23:28:10.569483 18014 tablet_bootstrap.cc:492] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: No bootstrap required, opened a new log
I20260926 23:28:10.569708 18017 tablet_bootstrap.cc:654] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:10.569806 18014 ts_tablet_manager.cc:1397] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260926 23:28:10.570422 18015 tablet_bootstrap.cc:492] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: No bootstrap required, opened a new log
I20260926 23:28:10.570669 18015 ts_tablet_manager.cc:1397] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 23:28:10.570835 18017 log.cc:824] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:10.573390 18017 tablet_bootstrap.cc:492] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: No bootstrap required, opened a new log
I20260926 23:28:10.573632 18017 ts_tablet_manager.cc:1397] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:28:10.580562 18014 raft_consensus.cc:348] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.581043 18014 raft_consensus.cc:374] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:10.581151 18014 raft_consensus.cc:729] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d655d433b14ad79950943f91e2da77, State: Initialized, Role: FOLLOWER
I20260926 23:28:10.581318 18015 raft_consensus.cc:348] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.581734 18015 raft_consensus.cc:374] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:10.581866 18015 raft_consensus.cc:729] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad1373ab900492e9811b14bdfc3c019, State: Initialized, Role: FOLLOWER
I20260926 23:28:10.581820 18014 consensus_queue.cc:260] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.581799 18017 raft_consensus.cc:348] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.582254 18017 raft_consensus.cc:374] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:10.582379 18017 raft_consensus.cc:729] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e77b42b2043467bb4480d027e1f07e9, State: Initialized, Role: FOLLOWER
I20260926 23:28:10.582600 18015 consensus_queue.cc:260] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.583151 17863 heartbeater.cc:503] Master 127.0.97.62:45947 was elected leader, sending a full tablet report...
I20260926 23:28:10.583043 18017 consensus_queue.cc:260] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.583904 18014 ts_tablet_manager.cc:1428] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 23:28:10.589663 17996 heartbeater.cc:503] Master 127.0.97.62:45947 was elected leader, sending a full tablet report...
I20260926 23:28:10.590636 18015 ts_tablet_manager.cc:1428] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Time spent starting tablet: real 0.020s user 0.019s sys 0.001s
I20260926 23:28:10.591269 18017 ts_tablet_manager.cc:1428] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
W20260926 23:28:10.664960 17730 tablet.cc:2369] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:10.673113 18021 raft_consensus.cc:482] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:28:10.673457 18021 raft_consensus.cc:504] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.675182 18021 leader_election.cc:288] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e77b42b2043467bb4480d027e1f07e9 (127.0.97.1:34971), b5d655d433b14ad79950943f91e2da77 (127.0.97.2:44351)
I20260926 23:28:10.681077 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.675745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44335 (local address 127.0.97.1:34971)
0926 23:28:10.675947 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.675955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.675979 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:28:10.676062 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.676071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.676193 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.676296 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:10.676800 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.677796 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.679370 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.680161 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.680328 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.680429 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.680708 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.680837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:28:10.681107 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.675364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34971 (local address 127.0.97.3:44335)
0926 23:28:10.675653 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.675676 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.675843 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.676329 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.676336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.676369 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:10.676624 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.676637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.677921 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.677929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.679211 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.679223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.680334 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.680340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.680529 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.680554 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.680658 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.680817 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 23:28:10.681816 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.675559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33545 (local address 127.0.97.2:44351)
0926 23:28:10.675785 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.675791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.676579 (+ 788us) server_negotiation.cc:407] Connection header received
0926 23:28:10.676810 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.676819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.676902 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.676991 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:10.677578 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.678272 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.679992 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.680818 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.680970 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.681065 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.681410 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.681558 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:28:10.681797 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.675499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44351 (local address 127.0.97.3:33545)
0926 23:28:10.676470 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.676484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.676633 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.677003 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.677008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.677029 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:10.677431 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.677446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.678418 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.678424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.679838 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.679850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.681022 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.681028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.681220 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.681267 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.681364 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.681495 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":7,"thread_start_us":548,"threads_started":1}
I20260926 23:28:10.682380 17684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779ddef499e5496ba913c85e164a1cb2" candidate_uuid: "4ad1373ab900492e9811b14bdfc3c019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e77b42b2043467bb4480d027e1f07e9" is_pre_election: true
I20260926 23:28:10.682896 17684 raft_consensus.cc:2454] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad1373ab900492e9811b14bdfc3c019 in term 0.
I20260926 23:28:10.682838 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779ddef499e5496ba913c85e164a1cb2" candidate_uuid: "4ad1373ab900492e9811b14bdfc3c019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d655d433b14ad79950943f91e2da77" is_pre_election: true
I20260926 23:28:10.683372 17818 raft_consensus.cc:2454] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad1373ab900492e9811b14bdfc3c019 in term 0.
W20260926 23:28:10.683544 17864 tablet.cc:2369] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:10.683646 18020 raft_consensus.cc:482] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:28:10.683693 17885 leader_election.cc:302] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad1373ab900492e9811b14bdfc3c019, 7e77b42b2043467bb4480d027e1f07e9; no voters:
I20260926 23:28:10.683889 18020 raft_consensus.cc:504] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.684140 18021 raft_consensus.cc:2781] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:28:10.684271 18021 raft_consensus.cc:482] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:28:10.684361 18021 raft_consensus.cc:3037] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:10.685791 18020 leader_election.cc:288] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e77b42b2043467bb4480d027e1f07e9 (127.0.97.1:34971), 4ad1373ab900492e9811b14bdfc3c019 (127.0.97.3:33331)
I20260926 23:28:10.687405 18021 raft_consensus.cc:504] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.688086 18021 leader_election.cc:288] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [CANDIDATE]: Term 1 election: Requested vote from peers 7e77b42b2043467bb4480d027e1f07e9 (127.0.97.1:34971), b5d655d433b14ad79950943f91e2da77 (127.0.97.2:44351)
I20260926 23:28:10.688454 17684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779ddef499e5496ba913c85e164a1cb2" candidate_uuid: "4ad1373ab900492e9811b14bdfc3c019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e77b42b2043467bb4480d027e1f07e9"
I20260926 23:28:10.688721 17684 raft_consensus.cc:3037] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:10.692201 17684 raft_consensus.cc:2454] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad1373ab900492e9811b14bdfc3c019 in term 1.
I20260926 23:28:10.692842 17885 leader_election.cc:302] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad1373ab900492e9811b14bdfc3c019, 7e77b42b2043467bb4480d027e1f07e9; no voters:
W20260926 23:28:10.692981 17997 tablet.cc:2369] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:10.693202 18021 raft_consensus.cc:2781] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:10.693157 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779ddef499e5496ba913c85e164a1cb2" candidate_uuid: "4ad1373ab900492e9811b14bdfc3c019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d655d433b14ad79950943f91e2da77"
I20260926 23:28:10.693387 17818 raft_consensus.cc:3037] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:10.696604 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.686679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41185 (local address 127.0.97.3:33331)
0926 23:28:10.686876 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.686882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.689178 (+ 2296us) server_negotiation.cc:407] Connection header received
0926 23:28:10.689430 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.689436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.689509 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.689599 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:10.690192 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.691313 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.694046 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.695361 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.695458 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.695566 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.696201 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.696343 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:28:10.697093 17818 raft_consensus.cc:2454] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad1373ab900492e9811b14bdfc3c019 in term 1.
I20260926 23:28:10.697546 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.686113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33331 (local address 127.0.97.2:41185)
0926 23:28:10.689063 (+ 2950us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.689077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.689269 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.689622 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.689626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.689645 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:10.690032 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.690043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.691457 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.691464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.693833 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.693852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.695793 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.695802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.695999 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.696033 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.696877 (+ 844us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.697031 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2844,"mutex_wait_us":22,"spinlock_wait_cycles":45696,"thread_start_us":2259,"threads_started":1}
I20260926 23:28:10.698385 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779ddef499e5496ba913c85e164a1cb2" candidate_uuid: "b5d655d433b14ad79950943f91e2da77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad1373ab900492e9811b14bdfc3c019" is_pre_election: true
I20260926 23:28:10.699004 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.686296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35289 (local address 127.0.97.1:34971)
0926 23:28:10.686509 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.686518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.688901 (+ 2383us) server_negotiation.cc:407] Connection header received
0926 23:28:10.689186 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.689192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.689283 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.689369 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:10.690463 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.691610 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.695779 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.697616 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.697738 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.697888 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.698619 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.698766 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 23:28:10.699323 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.685946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34971 (local address 127.0.97.2:35289)
0926 23:28:10.688796 (+ 2850us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:10.688817 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:10.689022 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:10.689752 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:10.689757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:10.689824 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:10.690302 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.690314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.691745 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.691752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.695518 (+ 3766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:10.695544 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.698132 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.698141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:10.698402 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.698453 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 23:28:10.698564 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:28:10.698753 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2669,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":2874,"threads_started":1}
I20260926 23:28:10.700067 17684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779ddef499e5496ba913c85e164a1cb2" candidate_uuid: "b5d655d433b14ad79950943f91e2da77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e77b42b2043467bb4480d027e1f07e9" is_pre_election: true
I20260926 23:28:10.700381 17684 raft_consensus.cc:2379] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b5d655d433b14ad79950943f91e2da77 in current term 1: Already voted for candidate 4ad1373ab900492e9811b14bdfc3c019 in this term.
I20260926 23:28:10.700947 18021 raft_consensus.cc:686] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 1 LEADER]: Becoming Leader. State: Replica: 4ad1373ab900492e9811b14bdfc3c019, State: Running, Role: LEADER
I20260926 23:28:10.701507 18021 consensus_queue.cc:237] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } }
I20260926 23:28:10.706082 17751 leader_election.cc:302] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5d655d433b14ad79950943f91e2da77; no voters: 4ad1373ab900492e9811b14bdfc3c019, 7e77b42b2043467bb4480d027e1f07e9
I20260926 23:28:10.706516 18020 raft_consensus.cc:2726] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 23:28:10.707041 17539 catalog_manager.cc:5495] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ad1373ab900492e9811b14bdfc3c019 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4ad1373ab900492e9811b14bdfc3c019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ad1373ab900492e9811b14bdfc3c019" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 } health_report { overall_health: UNKNOWN } } }
I20260926 23:28:10.762558 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.756769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.0.97.3:33331)
0926 23:28:10.756965 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.756971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.756990 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:28:10.757181 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.757185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.757308 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.757481 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:28:10.757877 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.758785 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.760235 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.760656 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.760872 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.761383 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:28:10.761927 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:28:10.762124 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.762173 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.762261 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:28:10.771183 17684 raft_consensus.cc:1261] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 1 FOLLOWER]: Refusing update from remote peer 4ad1373ab900492e9811b14bdfc3c019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:10.771251 17818 raft_consensus.cc:1261] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 1 FOLLOWER]: Refusing update from remote peer 4ad1373ab900492e9811b14bdfc3c019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:10.772099 18026 consensus_queue.cc:1035] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e77b42b2043467bb4480d027e1f07e9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:10.772500 18021 consensus_queue.cc:1035] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d655d433b14ad79950943f91e2da77" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:10.782560 18029 mvcc.cc:205] Tried to move back new op lower bound from 7333745830988009472 to 7333745830717755392. Current Snapshot: MvccSnapshot[applied={T|T < 7333745830988009472}]
I20260926 23:28:10.799037 17539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:10.799321 17539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:10.802726 17539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d404a68b2e94e4985ceac4da1e8ac87: Sending DeleteTablet for 3 replicas of tablet 779ddef499e5496ba913c85e164a1cb2
I20260926 23:28:10.803766 17931 tablet_service.cc:1514] Processing DeleteTablet for tablet 779ddef499e5496ba913c85e164a1cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34208
I20260926 23:28:10.803972 17664 tablet_service.cc:1514] Processing DeleteTablet for tablet 779ddef499e5496ba913c85e164a1cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51824
I20260926 23:28:10.804159 17798 tablet_service.cc:1514] Processing DeleteTablet for tablet 779ddef499e5496ba913c85e164a1cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45604
I20260926 23:28:10.806505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 17601
I20260926 23:28:10.815640 18040 tablet_replica.cc:330] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: stopping tablet replica
I20260926 23:28:10.816257 18040 raft_consensus.cc:2229] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:10.816668 18041 tablet_replica.cc:330] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: stopping tablet replica
I20260926 23:28:10.817036 18040 raft_consensus.cc:2258] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:10.817148 18041 raft_consensus.cc:2229] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:10.817579 18041 raft_consensus.cc:2258] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:10.818288 18039 tablet_replica.cc:330] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: stopping tablet replica
I20260926 23:28:10.818634 18039 raft_consensus.cc:2229] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:28:10.819222 18039 raft_consensus.cc:2258] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:10.820582 18041 ts_tablet_manager.cc:1905] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:10.821110 18039 ts_tablet_manager.cc:1905] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:10.822710 18040 ts_tablet_manager.cc:1905] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:10.828035 18039 ts_tablet_manager.cc:1918] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:10.828186 18039 log.cc:1196] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/779ddef499e5496ba913c85e164a1cb2
I20260926 23:28:10.828660 18039 ts_tablet_manager.cc:1939] T 779ddef499e5496ba913c85e164a1cb2 P 4ad1373ab900492e9811b14bdfc3c019: Deleting consensus metadata
I20260926 23:28:10.830394 17524 catalog_manager.cc:4841] TS 4ad1373ab900492e9811b14bdfc3c019 (127.0.97.3:33331): tablet 779ddef499e5496ba913c85e164a1cb2 (table test-table [id=15994b40ae934c54b994130ef6a11a0d]) successfully deleted
I20260926 23:28:10.834292 18041 ts_tablet_manager.cc:1918] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:10.834458 18041 log.cc:1196] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/779ddef499e5496ba913c85e164a1cb2
I20260926 23:28:10.834977 18041 ts_tablet_manager.cc:1939] T 779ddef499e5496ba913c85e164a1cb2 P b5d655d433b14ad79950943f91e2da77: Deleting consensus metadata
I20260926 23:28:10.835696 18040 ts_tablet_manager.cc:1918] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:10.835867 18040 log.cc:1196] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/779ddef499e5496ba913c85e164a1cb2
I20260926 23:28:10.836404 18040 ts_tablet_manager.cc:1939] T 779ddef499e5496ba913c85e164a1cb2 P 7e77b42b2043467bb4480d027e1f07e9: Deleting consensus metadata
I20260926 23:28:10.837009 17527 catalog_manager.cc:4841] TS b5d655d433b14ad79950943f91e2da77 (127.0.97.2:44351): tablet 779ddef499e5496ba913c85e164a1cb2 (table test-table [id=15994b40ae934c54b994130ef6a11a0d]) successfully deleted
I20260926 23:28:10.838406 17525 catalog_manager.cc:4841] TS 7e77b42b2043467bb4480d027e1f07e9 (127.0.97.1:34971): tablet 779ddef499e5496ba913c85e164a1cb2 (table test-table [id=15994b40ae934c54b994130ef6a11a0d]) successfully deleted
Sep 26 23:28:10 dist-test-slave-q5g8 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:28:10.848099 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.808300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.0.97.1:34971)
0926 23:28:10.808523 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.808532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.823455 (+ 14923us) server_negotiation.cc:407] Connection header received
0926 23:28:10.823903 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.823909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.823999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.824072 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:10.826078 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.827178 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.837220 (+ 10042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.837745 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.837827 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.842294 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:10.842305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:10.842319 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:10.842351 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:10.845348 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.846070 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.846078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.846080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.846212 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.846598 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.846608 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.846610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.846936 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:10.847262 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.847528 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.847750 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:28:10.848769 17724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:10.940588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 17601
I20260926 23:28:10.965435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 17735
Sep 26 23:28:10 dist-test-slave-q5g8 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:28:10.982889 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:10.966194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.0.97.2:44351)
0926 23:28:10.966381 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:10.966385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:10.966476 (+ 91us) server_negotiation.cc:407] Connection header received
0926 23:28:10.966654 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:10.966657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:10.966725 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:10.966808 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:10.968478 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.969252 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.971207 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:10.971597 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:10.971685 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:10.977062 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:10.977111 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:10.977150 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:10.977191 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:10.980062 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.980807 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.980815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.980817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.980944 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:10.981321 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:10.981327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:10.981330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:10.981703 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:10.982020 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:10.982324 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 23:28:10.982552 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 23:28:10.983502 17858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:11.075307 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 17735
I20260926 23:28:11.099145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 17868
Sep 26 23:28:11 dist-test-slave-q5g8 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:28:11.116489 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:11.099929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.0.97.3:33331)
0926 23:28:11.100114 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:11.100124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:11.100147 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:28:11.100283 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:11.100289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:11.100396 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:11.100514 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:11.101808 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:11.102963 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:11.104236 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:11.105005 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:11.105151 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:11.109440 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:11.109452 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:11.109473 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:11.109526 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:11.113601 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:11.114295 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:11.114306 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:11.114310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:11.114477 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:11.114849 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:11.114856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:11.114859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:11.115342 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:11.115658 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:11.115873 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 23:28:11.116114 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:28:11.117327 17991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:11.238710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 17868
I20260926 23:28:11.258456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 17508
I20260926 23:28:11.259574 17570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:11.373912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 17508
2026-09-26T23:28:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4341 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T23:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:11Z Disabled control of system clock
I20260926 23:28:11.450062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37051
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:28:11.775643 18050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:11.775934 18050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:11.776015 18050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:11.783310 18050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:11.783417 18050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:11.783455 18050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:11.783504 18050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:11.783548 18050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:11.792701 18050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:11.794356 18050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:11.796497 18050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:11.810202 18055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:11.810235 18056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:11.810357 18058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:11.810920 18050 server_base.cc:1034] running on GCE node
I20260926 23:28:11.811708 18050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:11.813140 18050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:11.814335 18050 hybrid_clock.cc:648] HybridClock initialized: now 1790465291814297 us; error 30 us; skew 500 ppm
I20260926 23:28:11.814680 18050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:11.820289 18050 webserver.cc:460] Webserver started at http://127.0.97.62:34179/ using document root <none> and password file <none>
I20260926 23:28:11.821077 18050 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:11.821223 18050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:11.821516 18050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:11.824295 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "11c1a1cdfd934047912c2c9e04a96412"
format_stamp: "Formatted at 2026-09-26 23:28:11 on dist-test-slave-q5g8"
server_key: "de0245f04cc583384aec6b006e3b75fc"
server_key_iv: "b2dd69135074d84111b022e6d8a2a059"
server_key_version: "encryptionkey@0"
I20260926 23:28:11.825083 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "11c1a1cdfd934047912c2c9e04a96412"
format_stamp: "Formatted at 2026-09-26 23:28:11 on dist-test-slave-q5g8"
server_key: "de0245f04cc583384aec6b006e3b75fc"
server_key_iv: "b2dd69135074d84111b022e6d8a2a059"
server_key_version: "encryptionkey@0"
I20260926 23:28:11.829943 18050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 23:28:11.833304 18064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:11.834604 18050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:28:11.834793 18050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "11c1a1cdfd934047912c2c9e04a96412"
format_stamp: "Formatted at 2026-09-26 23:28:11 on dist-test-slave-q5g8"
server_key: "de0245f04cc583384aec6b006e3b75fc"
server_key_iv: "b2dd69135074d84111b022e6d8a2a059"
server_key_version: "encryptionkey@0"
I20260926 23:28:11.835009 18050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:11.849939 18050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:11.850725 18050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:11.851006 18050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:11.869246 18050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37051
I20260926 23:28:11.869277 18115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37051 every 8 connection(s)
I20260926 23:28:11.870569 18050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:11.872219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18050
I20260926 23:28:11.872403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:11.872912 388 external_mini_cluster.cc:1428] Setting key f4286fda66efa91260c6412a44115fd6
I20260926 23:28:11.874907 18116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:11.888221 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:11.874854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.0.97.62:37051)
0926 23:28:11.875954 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:11.875969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:11.876018 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:28:11.878063 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:11.878089 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:11.878696 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:11.879123 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:11.879674 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:11.881744 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:11.883070 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:11.884088 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:11.884261 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:11.885378 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:11.885390 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:11.885412 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:11.886166 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:11.886269 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:11.886300 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:11.886423 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:11.886635 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 23:28:11.886978 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":357,"threads_started":1}
I20260926 23:28:11.889534 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412: Bootstrap starting.
I20260926 23:28:11.892506 18116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:11.893492 18116 log.cc:824] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:11.896436 18116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412: No bootstrap required, opened a new log
I20260926 23:28:11.903813 18116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1a1cdfd934047912c2c9e04a96412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260926 23:28:11.904134 18116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:11.904229 18116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c1a1cdfd934047912c2c9e04a96412, State: Initialized, Role: FOLLOWER
I20260926 23:28:11.904857 18116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1a1cdfd934047912c2c9e04a96412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260926 23:28:11.905095 18116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:11.905222 18116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:11.905393 18116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:11.908233 18116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1a1cdfd934047912c2c9e04a96412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260926 23:28:11.908790 18116 leader_election.cc:302] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c1a1cdfd934047912c2c9e04a96412; no voters:
I20260926 23:28:11.909379 18116 leader_election.cc:288] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:11.909536 18121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:11.910265 18121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [term 1 LEADER]: Becoming Leader. State: Replica: 11c1a1cdfd934047912c2c9e04a96412, State: Running, Role: LEADER
I20260926 23:28:11.910763 18121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1a1cdfd934047912c2c9e04a96412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260926 23:28:11.911491 18116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:11.913358 18123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c1a1cdfd934047912c2c9e04a96412. Latest consensus state: current_term: 1 leader_uuid: "11c1a1cdfd934047912c2c9e04a96412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1a1cdfd934047912c2c9e04a96412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } } }
I20260926 23:28:11.913781 18123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:11.914510 18122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c1a1cdfd934047912c2c9e04a96412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1a1cdfd934047912c2c9e04a96412" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } } }
I20260926 23:28:11.914779 18122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:11.914835 18128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:11.919870 18128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:11.929368 18128 catalog_manager.cc:1347] Generated new cluster ID: 59723b36ed3943cd9b7588a3e37f978e
I20260926 23:28:11.929481 18128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:11.959807 18128 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:11.961331 18128 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:11.967882 18128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412: Generated new TSK 0
I20260926 23:28:11.968930 18128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:11.981390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37051
--builtin_ntp_servers=127.0.97.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:28:12.329849 18140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:12.330147 18140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:12.330327 18140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:12.338254 18140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:12.338400 18140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:12.338580 18140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:28:12.348781 18140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:12.350157 18140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:12.352039 18140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:12.367065 18145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:12.367195 18148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:12.367089 18146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:12.367792 18140 server_base.cc:1034] running on GCE node
I20260926 23:28:12.368446 18140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:12.369493 18140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:12.370743 18140 hybrid_clock.cc:648] HybridClock initialized: now 1790465292370698 us; error 31 us; skew 500 ppm
I20260926 23:28:12.371122 18140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:12.373723 18140 webserver.cc:460] Webserver started at http://127.0.97.1:41949/ using document root <none> and password file <none>
I20260926 23:28:12.374477 18140 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:12.374631 18140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:12.374977 18140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:12.377986 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "155b0f4a82c4477c9c6bdc20bf293aa5"
format_stamp: "Formatted at 2026-09-26 23:28:12 on dist-test-slave-q5g8"
server_key: "755fb79b297e0005310121e793a48b22"
server_key_iv: "7359c49e5de58e1f576974b23a5584ff"
server_key_version: "encryptionkey@0"
I20260926 23:28:12.378998 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "155b0f4a82c4477c9c6bdc20bf293aa5"
format_stamp: "Formatted at 2026-09-26 23:28:12 on dist-test-slave-q5g8"
server_key: "755fb79b297e0005310121e793a48b22"
server_key_iv: "7359c49e5de58e1f576974b23a5584ff"
server_key_version: "encryptionkey@0"
I20260926 23:28:12.384374 18140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:28:12.387980 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:12.389705 18140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:28:12.389830 18140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "155b0f4a82c4477c9c6bdc20bf293aa5"
format_stamp: "Formatted at 2026-09-26 23:28:12 on dist-test-slave-q5g8"
server_key: "755fb79b297e0005310121e793a48b22"
server_key_iv: "7359c49e5de58e1f576974b23a5584ff"
server_key_version: "encryptionkey@0"
I20260926 23:28:12.390002 18140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:12.414601 18140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:12.415436 18140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:12.415755 18140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:12.416857 18140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:12.418390 18140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:12.418484 18140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:12.418576 18140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:12.418644 18140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:12.455499 18140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38503
I20260926 23:28:12.455559 18266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38503 every 8 connection(s)
I20260926 23:28:12.456985 18140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:28:12.459225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18140
I20260926 23:28:12.459407 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:28:12.459930 388 external_mini_cluster.cc:1428] Setting key 5f759db103542a2f1b2b0bcdb98ea108
I20260926 23:28:12.463563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37051
--builtin_ntp_servers=127.0.97.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:28:12.473452 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:12.459901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59085 (local address 127.0.97.62:37051)
0926 23:28:12.461045 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:12.461053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:12.461897 (+ 844us) server_negotiation.cc:407] Connection header received
0926 23:28:12.464511 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:12.464515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:12.464580 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:12.464676 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:12.466215 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.466953 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.469807 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.470246 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.470457 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:12.472310 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:12.472315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:12.472318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:12.472339 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:12.472384 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:12.472396 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:12.472464 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:12.472856 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 23:28:12.472975 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":839,"threads_started":1}
I20260926 23:28:12.474428 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:12.460476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37051 (local address 127.0.97.1:59085)
0926 23:28:12.461712 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:12.461766 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:12.464197 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:12.465000 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:12.465014 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:12.465167 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:12.466019 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:12.466048 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.467210 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.467219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:12.469630 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:12.469653 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.470579 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.470590 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:12.470965 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:12.471836 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:12.471842 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:12.471902 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:12.472032 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:12.472512 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:12.472527 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:12.472540 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 23:28:12.472801 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 23:28:12.473151 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":429,"threads_started":1}
I20260926 23:28:12.476280 18267 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37051
I20260926 23:28:12.476642 18267 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:12.477473 18267 heartbeater.cc:511] Master 127.0.97.62:37051 requested a full tablet report, sending...
I20260926 23:28:12.479662 18081 ts_manager.cc:194] Registered new tserver with Master: 155b0f4a82c4477c9c6bdc20bf293aa5 (127.0.97.1:38503)
I20260926 23:28:12.481320 18081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59085
W20260926 23:28:12.791996 18271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:12.792255 18271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:12.792390 18271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:12.799770 18271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:12.799861 18271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:12.800036 18271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:28:12.808977 18271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:12.810258 18271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:12.812085 18271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:12.825143 18276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:12.825341 18279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:12.825421 18277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:12.827073 18271 server_base.cc:1034] running on GCE node
I20260926 23:28:12.827672 18271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:12.828608 18271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:12.829825 18271 hybrid_clock.cc:648] HybridClock initialized: now 1790465292829777 us; error 40 us; skew 500 ppm
I20260926 23:28:12.830181 18271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:12.832520 18271 webserver.cc:460] Webserver started at http://127.0.97.2:37441/ using document root <none> and password file <none>
I20260926 23:28:12.833227 18271 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:12.833323 18271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:12.833679 18271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:12.836519 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "84c64dd674ba40618bd92452023ed91a"
format_stamp: "Formatted at 2026-09-26 23:28:12 on dist-test-slave-q5g8"
server_key: "5ba562c8a21523008f4dd48442c3a0d8"
server_key_iv: "10b1c81d6e710f933b39fa997531b613"
server_key_version: "encryptionkey@0"
I20260926 23:28:12.837357 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "84c64dd674ba40618bd92452023ed91a"
format_stamp: "Formatted at 2026-09-26 23:28:12 on dist-test-slave-q5g8"
server_key: "5ba562c8a21523008f4dd48442c3a0d8"
server_key_iv: "10b1c81d6e710f933b39fa997531b613"
server_key_version: "encryptionkey@0"
I20260926 23:28:12.842587 18271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 23:28:12.845978 18285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:12.847229 18271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:28:12.847389 18271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "84c64dd674ba40618bd92452023ed91a"
format_stamp: "Formatted at 2026-09-26 23:28:12 on dist-test-slave-q5g8"
server_key: "5ba562c8a21523008f4dd48442c3a0d8"
server_key_iv: "10b1c81d6e710f933b39fa997531b613"
server_key_version: "encryptionkey@0"
I20260926 23:28:12.847597 18271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:12.868873 18271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:12.869683 18271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:12.869966 18271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:12.870949 18271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:12.872565 18271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:12.872684 18271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:12.872826 18271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:12.872915 18271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:12.906467 18271 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44591
I20260926 23:28:12.906553 18397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44591 every 8 connection(s)
I20260926 23:28:12.907738 18271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:28:12.911237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18271
I20260926 23:28:12.911424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:28:12.912014 388 external_mini_cluster.cc:1428] Setting key 718f48e2883f092aa567feae68e98af2
I20260926 23:28:12.914984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37051
--builtin_ntp_servers=127.0.97.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:28:12.921837 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:12.910421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33139 (local address 127.0.97.62:37051)
0926 23:28:12.910633 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:12.910640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:12.912216 (+ 1576us) server_negotiation.cc:407] Connection header received
0926 23:28:12.914785 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:12.914793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:12.914897 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:12.915018 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:12.916132 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.917011 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.918695 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.919120 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.919267 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:12.920766 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:12.920771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:12.920775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:12.920795 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:12.920837 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:12.920851 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:12.920954 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:12.921328 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 23:28:12.921434 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:28:12.922850 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:12.910806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37051 (local address 127.0.97.2:33139)
0926 23:28:12.912037 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:12.912107 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:12.914503 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:12.915255 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:12.915264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:12.915356 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:12.915940 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:12.915958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.917185 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.917192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:12.918520 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:12.918533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:12.919360 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:12.919368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:12.919626 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:12.920335 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:12.920338 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:12.920404 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:12.920510 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:12.920965 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:12.920979 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:12.920989 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:28:12.921317 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 23:28:12.921741 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":319,"threads_started":1}
I20260926 23:28:12.924364 18398 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37051
I20260926 23:28:12.924665 18398 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:12.925557 18398 heartbeater.cc:511] Master 127.0.97.62:37051 requested a full tablet report, sending...
I20260926 23:28:12.927132 18081 ts_manager.cc:194] Registered new tserver with Master: 84c64dd674ba40618bd92452023ed91a (127.0.97.2:44591)
I20260926 23:28:12.928319 18081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33139
W20260926 23:28:13.240306 18401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:13.240567 18401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:13.240655 18401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:13.247941 18401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:13.248035 18401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:13.248142 18401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:28:13.257217 18401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:13.258383 18401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:13.260039 18401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:13.273138 18406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:13.273170 18409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:13.273203 18407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:13.274158 18401 server_base.cc:1034] running on GCE node
I20260926 23:28:13.274705 18401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:13.275588 18401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:13.276831 18401 hybrid_clock.cc:648] HybridClock initialized: now 1790465293276788 us; error 59 us; skew 500 ppm
I20260926 23:28:13.277204 18401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:13.279485 18401 webserver.cc:460] Webserver started at http://127.0.97.3:37047/ using document root <none> and password file <none>
I20260926 23:28:13.280165 18401 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:13.280296 18401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:13.280615 18401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:13.283412 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "c95242bbcc2f4ab5ad285e953bc13850"
format_stamp: "Formatted at 2026-09-26 23:28:13 on dist-test-slave-q5g8"
server_key: "51aa72454ddc6f39274206e222dc2a99"
server_key_iv: "819e64b624e288cfeed33b73c02f64ac"
server_key_version: "encryptionkey@0"
I20260926 23:28:13.284184 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c95242bbcc2f4ab5ad285e953bc13850"
format_stamp: "Formatted at 2026-09-26 23:28:13 on dist-test-slave-q5g8"
server_key: "51aa72454ddc6f39274206e222dc2a99"
server_key_iv: "819e64b624e288cfeed33b73c02f64ac"
server_key_version: "encryptionkey@0"
I20260926 23:28:13.289143 18401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:28:13.292536 18415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:13.294030 18401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:28:13.294209 18401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "c95242bbcc2f4ab5ad285e953bc13850"
format_stamp: "Formatted at 2026-09-26 23:28:13 on dist-test-slave-q5g8"
server_key: "51aa72454ddc6f39274206e222dc2a99"
server_key_iv: "819e64b624e288cfeed33b73c02f64ac"
server_key_version: "encryptionkey@0"
I20260926 23:28:13.294440 18401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:13.307773 18401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:13.308524 18401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:13.308881 18401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:13.309805 18401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:13.311208 18401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:13.311326 18401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:13.311460 18401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:13.311546 18401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:13.346926 18401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42183
I20260926 23:28:13.347011 18527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42183 every 8 connection(s)
I20260926 23:28:13.348218 18401 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:28:13.352262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18401
I20260926 23:28:13.352450 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:28:13.353031 388 external_mini_cluster.cc:1428] Setting key 7b80586f67f645130d682cc808f600b3
I20260926 23:28:13.361406 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.351038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54157 (local address 127.0.97.62:37051)
0926 23:28:13.351257 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.351264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.352950 (+ 1686us) server_negotiation.cc:407] Connection header received
0926 23:28:13.354903 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.354908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.355017 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.355126 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:13.356266 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.357186 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.358674 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.359113 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.359244 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.360550 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:13.360556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:13.360561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:13.360580 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:13.360622 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:13.360635 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:13.360770 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.360988 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.361080 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:28:13.362300 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.351660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37051 (local address 127.0.97.3:54157)
0926 23:28:13.352810 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:13.352845 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:13.354638 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:13.355372 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:13.355379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:13.355462 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:13.356110 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.356125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.357328 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.357334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.358529 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.358540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.359330 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.359336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.359558 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.360256 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:13.360259 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:13.360293 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:13.360362 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:13.360771 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:13.360783 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:13.360793 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:28:13.360963 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 23:28:13.361323 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":371,"threads_started":1}
I20260926 23:28:13.363651 18528 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37051
I20260926 23:28:13.364055 18528 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:13.364936 18528 heartbeater.cc:511] Master 127.0.97.62:37051 requested a full tablet report, sending...
I20260926 23:28:13.366377 18081 ts_manager.cc:194] Registered new tserver with Master: c95242bbcc2f4ab5ad285e953bc13850 (127.0.97.3:42183)
I20260926 23:28:13.367617 18081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54157
I20260926 23:28:13.368145 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:28:13.377928 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.372276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.0.97.62:37051)
0926 23:28:13.372518 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.372523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.372843 (+ 320us) server_negotiation.cc:407] Connection header received
0926 23:28:13.372978 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.372982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.373045 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.373122 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:13.373597 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.374267 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.375785 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.376179 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.376263 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.377274 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:13.377278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:13.377281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:13.377298 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:13.377328 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:13.377339 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:13.377407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.377527 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.377595 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:28:13.381722 18081 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:28:13.389195 18081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:28:13.435132 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.402582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42183 (local address 127.0.0.1:55402)
0926 23:28:13.425108 (+ 22526us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:13.425134 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:13.425296 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:13.426907 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:13.426914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:13.426944 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:13.427439 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.427453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.429492 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.429502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.432281 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.432306 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.434166 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.434172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.434399 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.434454 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 23:28:13.434618 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 23:28:13.434796 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22308,"thread_start_us":22827,"threads_started":1}
I20260926 23:28:13.435165 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.403248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.0.97.3:42183)
0926 23:28:13.426184 (+ 22936us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.426193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.426232 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:28:13.426341 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.426351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.426645 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.427633 (+ 988us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:13.427846 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.429310 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.432521 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.433964 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.434111 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.434411 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.434654 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.434821 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22701,"thread_start_us":22536,"threads_started":1}
I20260926 23:28:13.436975 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.402285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38503 (local address 127.0.0.1:39574)
0926 23:28:13.422913 (+ 20628us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:13.422968 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:13.423237 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:13.429684 (+ 6447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:13.429689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:13.429723 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:13.430230 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.430243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.432093 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.432099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.434093 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.434108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.436147 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.436153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.436337 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.436368 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:28:13.436451 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 23:28:13.436571 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20276,"mutex_wait_us":3,"thread_start_us":26737,"threads_started":1}
I20260926 23:28:13.437880 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.402461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.97.1:38503)
0926 23:28:13.429084 (+ 26623us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.429094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.429121 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:28:13.429229 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.429236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.429472 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.429661 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:13.430445 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.431931 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.434341 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.435958 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.436109 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.436342 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.437297 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.437474 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26471,"thread_start_us":26359,"threads_started":1}
I20260926 23:28:13.438074 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.406245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.0.97.2:44591)
0926 23:28:13.424953 (+ 18708us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.424965 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.428552 (+ 3587us) server_negotiation.cc:407] Connection header received
0926 23:28:13.428732 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.428741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.428987 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.429173 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:13.429957 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.431374 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.433872 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.436761 (+ 2889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.436914 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.437182 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.437591 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.437791 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18253,"thread_start_us":18425,"threads_started":1}
I20260926 23:28:13.438444 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.405510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44591 (local address 127.0.0.1:46818)
0926 23:28:13.428384 (+ 22874us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:13.428406 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:13.428557 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:13.429234 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:13.429238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:13.429261 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:13.429761 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.429778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.431532 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.431538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.433668 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.433682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.436998 (+ 3316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.437005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.437391 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.437427 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:28:13.437529 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 23:28:13.437683 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22728,"mutex_wait_us":1,"thread_start_us":16861,"threads_started":1}
I20260926 23:28:13.439674 18463 tablet_service.cc:1467] Processing CreateTablet for tablet fdf49b411946433eb0df7015d7712a3a (DEFAULT_TABLE table=test-table [id=98f8203e290e450380145f08e7a5a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:13.441048 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf49b411946433eb0df7015d7712a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:13.441869 18202 tablet_service.cc:1467] Processing CreateTablet for tablet fdf49b411946433eb0df7015d7712a3a (DEFAULT_TABLE table=test-table [id=98f8203e290e450380145f08e7a5a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:13.441911 18333 tablet_service.cc:1467] Processing CreateTablet for tablet fdf49b411946433eb0df7015d7712a3a (DEFAULT_TABLE table=test-table [id=98f8203e290e450380145f08e7a5a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:13.443305 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf49b411946433eb0df7015d7712a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:13.443305 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf49b411946433eb0df7015d7712a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:13.452529 18546 tablet_bootstrap.cc:492] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Bootstrap starting.
I20260926 23:28:13.455938 18546 tablet_bootstrap.cc:654] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:13.458217 18546 log.cc:824] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:13.465963 18549 tablet_bootstrap.cc:492] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Bootstrap starting.
I20260926 23:28:13.466708 18547 tablet_bootstrap.cc:492] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Bootstrap starting.
I20260926 23:28:13.470630 18547 tablet_bootstrap.cc:654] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:13.471989 18549 tablet_bootstrap.cc:654] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:13.471988 18547 log.cc:824] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:13.473208 18549 log.cc:824] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:13.473306 18546 tablet_bootstrap.cc:492] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: No bootstrap required, opened a new log
I20260926 23:28:13.473582 18546 ts_tablet_manager.cc:1397] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20260926 23:28:13.475018 18547 tablet_bootstrap.cc:492] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: No bootstrap required, opened a new log
I20260926 23:28:13.475286 18547 ts_tablet_manager.cc:1397] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:28:13.475898 18549 tablet_bootstrap.cc:492] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: No bootstrap required, opened a new log
I20260926 23:28:13.476155 18549 ts_tablet_manager.cc:1397] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 23:28:13.484102 18549 raft_consensus.cc:348] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.484279 18546 raft_consensus.cc:348] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.484570 18267 heartbeater.cc:503] Master 127.0.97.62:37051 was elected leader, sending a full tablet report...
I20260926 23:28:13.484608 18549 raft_consensus.cc:374] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:13.484730 18549 raft_consensus.cc:729] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155b0f4a82c4477c9c6bdc20bf293aa5, State: Initialized, Role: FOLLOWER
I20260926 23:28:13.484735 18546 raft_consensus.cc:374] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:13.484833 18546 raft_consensus.cc:729] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95242bbcc2f4ab5ad285e953bc13850, State: Initialized, Role: FOLLOWER
I20260926 23:28:13.485432 18546 consensus_queue.cc:260] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.485386 18549 consensus_queue.cc:260] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.485671 18547 raft_consensus.cc:348] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.486125 18547 raft_consensus.cc:374] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:13.486251 18547 raft_consensus.cc:729] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c64dd674ba40618bd92452023ed91a, State: Initialized, Role: FOLLOWER
I20260926 23:28:13.487064 18547 consensus_queue.cc:260] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.494102 18528 heartbeater.cc:503] Master 127.0.97.62:37051 was elected leader, sending a full tablet report...
I20260926 23:28:13.494846 18549 ts_tablet_manager.cc:1428] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Time spent starting tablet: real 0.019s user 0.009s sys 0.012s
I20260926 23:28:13.494994 18547 ts_tablet_manager.cc:1428] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260926 23:28:13.495362 18398 heartbeater.cc:503] Master 127.0.97.62:37051 was elected leader, sending a full tablet report...
I20260926 23:28:13.496050 18546 ts_tablet_manager.cc:1428] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s
W20260926 23:28:13.599956 18529 tablet.cc:2369] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:13.633328 18552 raft_consensus.cc:482] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:28:13.633648 18552 raft_consensus.cc:504] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.635272 18552 leader_election.cc:288] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155b0f4a82c4477c9c6bdc20bf293aa5 (127.0.97.1:38503), 84c64dd674ba40618bd92452023ed91a (127.0.97.2:44591)
I20260926 23:28:13.641754 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.635697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58201 (local address 127.0.97.2:44591)
0926 23:28:13.635909 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.635921 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.636021 (+ 100us) server_negotiation.cc:407] Connection header received
0926 23:28:13.636250 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.636254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.636323 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.636422 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:13.636956 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.637975 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.639894 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.640969 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.641060 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.641124 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.641474 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.641559 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:28:13.641894 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.635710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44591 (local address 127.0.97.3:58201)
0926 23:28:13.635905 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:13.635924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:13.636103 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:13.636507 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:13.636512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:13.636545 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:13.636761 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.636772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.638107 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.638112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.639743 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.639754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.641108 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.641116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.641314 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.641352 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:28:13.641447 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 23:28:13.641583 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 23:28:13.642869 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.635710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38503 (local address 127.0.97.3:50083)
0926 23:28:13.636735 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:13.636756 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:13.636861 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:13.637173 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:13.637177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:13.637223 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:13.637549 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.637560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.638696 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.638737 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.640550 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:13.640565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.642256 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.642262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:13.642424 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.642462 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:28:13.642552 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 23:28:13.642641 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":222,"thread_start_us":407,"threads_started":1}
I20260926 23:28:13.642997 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.635697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50083 (local address 127.0.97.1:38503)
0926 23:28:13.635905 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.635913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.636842 (+ 929us) server_negotiation.cc:407] Connection header received
0926 23:28:13.636996 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.636999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.637070 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.637136 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:13.637688 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.638555 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.640798 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.641935 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.642061 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.642163 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.642618 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.642753 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:28:13.642982 18353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf49b411946433eb0df7015d7712a3a" candidate_uuid: "c95242bbcc2f4ab5ad285e953bc13850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c64dd674ba40618bd92452023ed91a" is_pre_election: true
I20260926 23:28:13.643565 18353 raft_consensus.cc:2454] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95242bbcc2f4ab5ad285e953bc13850 in term 0.
I20260926 23:28:13.643878 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf49b411946433eb0df7015d7712a3a" candidate_uuid: "c95242bbcc2f4ab5ad285e953bc13850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" is_pre_election: true
I20260926 23:28:13.644261 18419 leader_election.cc:302] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c64dd674ba40618bd92452023ed91a, c95242bbcc2f4ab5ad285e953bc13850; no voters:
I20260926 23:28:13.644482 18222 raft_consensus.cc:2454] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95242bbcc2f4ab5ad285e953bc13850 in term 0.
I20260926 23:28:13.644707 18552 raft_consensus.cc:2781] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:28:13.644874 18552 raft_consensus.cc:482] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:28:13.645012 18552 raft_consensus.cc:3037] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:13.647132 18552 raft_consensus.cc:504] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.647668 18552 leader_election.cc:288] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [CANDIDATE]: Term 1 election: Requested vote from peers 155b0f4a82c4477c9c6bdc20bf293aa5 (127.0.97.1:38503), 84c64dd674ba40618bd92452023ed91a (127.0.97.2:44591)
I20260926 23:28:13.648088 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf49b411946433eb0df7015d7712a3a" candidate_uuid: "c95242bbcc2f4ab5ad285e953bc13850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5"
I20260926 23:28:13.648089 18353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf49b411946433eb0df7015d7712a3a" candidate_uuid: "c95242bbcc2f4ab5ad285e953bc13850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c64dd674ba40618bd92452023ed91a"
I20260926 23:28:13.648270 18222 raft_consensus.cc:3037] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:13.648299 18353 raft_consensus.cc:3037] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:13.650503 18222 raft_consensus.cc:2454] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95242bbcc2f4ab5ad285e953bc13850 in term 1.
I20260926 23:28:13.650609 18353 raft_consensus.cc:2454] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95242bbcc2f4ab5ad285e953bc13850 in term 1.
I20260926 23:28:13.650923 18418 leader_election.cc:302] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155b0f4a82c4477c9c6bdc20bf293aa5, c95242bbcc2f4ab5ad285e953bc13850; no voters:
I20260926 23:28:13.651240 18552 raft_consensus.cc:2781] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:13.651728 18552 raft_consensus.cc:686] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 1 LEADER]: Becoming Leader. State: Replica: c95242bbcc2f4ab5ad285e953bc13850, State: Running, Role: LEADER
I20260926 23:28:13.652248 18552 consensus_queue.cc:237] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } }
I20260926 23:28:13.657120 18081 catalog_manager.cc:5495] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c95242bbcc2f4ab5ad285e953bc13850 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c95242bbcc2f4ab5ad285e953bc13850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95242bbcc2f4ab5ad285e953bc13850" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 } health_report { overall_health: UNKNOWN } } }
W20260926 23:28:13.659722 18399 tablet.cc:2369] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:28:13.709502 18268 tablet.cc:2369] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:13.723696 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.718513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.0.97.3:42183)
0926 23:28:13.718677 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.718681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.718800 (+ 119us) server_negotiation.cc:407] Connection header received
0926 23:28:13.718972 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.718976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.719063 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.719205 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:28:13.719491 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.720308 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.721844 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.722266 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.722358 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.722764 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:28:13.723260 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:28:13.723363 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.723450 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.723510 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:28:13.731925 18353 raft_consensus.cc:1261] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 1 FOLLOWER]: Refusing update from remote peer c95242bbcc2f4ab5ad285e953bc13850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:13.731925 18222 raft_consensus.cc:1261] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 1 FOLLOWER]: Refusing update from remote peer c95242bbcc2f4ab5ad285e953bc13850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:13.732983 18556 consensus_queue.cc:1035] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155b0f4a82c4477c9c6bdc20bf293aa5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:13.733379 18552 consensus_queue.cc:1035] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c64dd674ba40618bd92452023ed91a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:13.744267 18560 mvcc.cc:205] Tried to move back new op lower bound from 7333745843115393024 to 7333745842803933184. Current Snapshot: MvccSnapshot[applied={T|T < 7333745843115393024}]
I20260926 23:28:13.758508 18081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:13.758777 18081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:13.762100 18081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11c1a1cdfd934047912c2c9e04a96412: Sending DeleteTablet for 3 replicas of tablet fdf49b411946433eb0df7015d7712a3a
I20260926 23:28:13.763207 18202 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf49b411946433eb0df7015d7712a3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:13 UTC) from {username='slave'} at 127.0.0.1:39574
I20260926 23:28:13.763218 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf49b411946433eb0df7015d7712a3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:13 UTC) from {username='slave'} at 127.0.0.1:55402
I20260926 23:28:13.763926 18333 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf49b411946433eb0df7015d7712a3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:13 UTC) from {username='slave'} at 127.0.0.1:46818
I20260926 23:28:13.765133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18140
I20260926 23:28:13.779281 18570 tablet_replica.cc:330] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: stopping tablet replica
I20260926 23:28:13.779284 18571 tablet_replica.cc:330] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: stopping tablet replica
I20260926 23:28:13.779845 18571 raft_consensus.cc:2229] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:13.780093 18569 tablet_replica.cc:330] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: stopping tablet replica
I20260926 23:28:13.780290 18570 raft_consensus.cc:2229] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:13.780411 18571 raft_consensus.cc:2258] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:13.780825 18570 raft_consensus.cc:2258] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:13.780821 18569 raft_consensus.cc:2229] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:28:13.782245 18569 raft_consensus.cc:2258] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:13.782835 18571 ts_tablet_manager.cc:1905] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:13.782835 18570 ts_tablet_manager.cc:1905] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:13.788264 18569 ts_tablet_manager.cc:1905] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:13.790004 18571 ts_tablet_manager.cc:1918] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:13.790002 18570 ts_tablet_manager.cc:1918] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:13.790172 18570 log.cc:1196] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/fdf49b411946433eb0df7015d7712a3a
I20260926 23:28:13.790172 18571 log.cc:1196] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/fdf49b411946433eb0df7015d7712a3a
I20260926 23:28:13.790720 18571 ts_tablet_manager.cc:1939] T fdf49b411946433eb0df7015d7712a3a P 84c64dd674ba40618bd92452023ed91a: Deleting consensus metadata
I20260926 23:28:13.790720 18570 ts_tablet_manager.cc:1939] T fdf49b411946433eb0df7015d7712a3a P 155b0f4a82c4477c9c6bdc20bf293aa5: Deleting consensus metadata
I20260926 23:28:13.792642 18068 catalog_manager.cc:4841] TS 84c64dd674ba40618bd92452023ed91a (127.0.97.2:44591): tablet fdf49b411946433eb0df7015d7712a3a (table test-table [id=98f8203e290e450380145f08e7a5a0bd]) successfully deleted
I20260926 23:28:13.792987 18067 catalog_manager.cc:4841] TS 155b0f4a82c4477c9c6bdc20bf293aa5 (127.0.97.1:38503): tablet fdf49b411946433eb0df7015d7712a3a (table test-table [id=98f8203e290e450380145f08e7a5a0bd]) successfully deleted
I20260926 23:28:13.795668 18569 ts_tablet_manager.cc:1918] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:13.795838 18569 log.cc:1196] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/fdf49b411946433eb0df7015d7712a3a
I20260926 23:28:13.796378 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.766273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.0.97.1:38503)
0926 23:28:13.766523 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.766532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.781679 (+ 15147us) server_negotiation.cc:407] Connection header received
0926 23:28:13.782205 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.782212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.782325 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.782407 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:13.784810 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.785983 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.792731 (+ 6748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.793374 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.793507 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.795015 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:13.795023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:13.795036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:13.795602 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:13.795649 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:13.795668 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:13.795812 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.795919 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.796031 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 23:28:13.796542 18569 ts_tablet_manager.cc:1939] T fdf49b411946433eb0df7015d7712a3a P c95242bbcc2f4ab5ad285e953bc13850: Deleting consensus metadata
I20260926 23:28:13.797086 18262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:13.798406 18066 catalog_manager.cc:4841] TS c95242bbcc2f4ab5ad285e953bc13850 (127.0.97.3:42183): tablet fdf49b411946433eb0df7015d7712a3a (table test-table [id=98f8203e290e450380145f08e7a5a0bd]) successfully deleted
I20260926 23:28:13.927538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18140
I20260926 23:28:13.946537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18271
I20260926 23:28:13.954916 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:13.947484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.0.97.2:44591)
0926 23:28:13.947709 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:13.947716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:13.947738 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:28:13.947838 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:13.947844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:13.947960 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:13.948087 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:13.948504 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.949368 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.950756 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:13.951326 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:13.951479 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:13.953156 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:13.953169 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:13.953187 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:13.953939 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:13.954003 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:13.954030 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:13.954214 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:13.954360 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 23:28:13.954502 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:28:13.956202 18393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:14.054934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18271
I20260926 23:28:14.073673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18401
I20260926 23:28:14.082365 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:14.074593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.0.97.3:42183)
0926 23:28:14.074839 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:14.074849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:14.074950 (+ 101us) server_negotiation.cc:407] Connection header received
0926 23:28:14.075203 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:14.075208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:14.075305 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:14.075457 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:14.075936 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:14.076955 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:14.078416 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:14.078922 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:14.079082 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:14.080504 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:14.080516 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:14.080533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:14.081449 (+ 916us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:14.081512 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:14.081534 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:14.081684 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:14.081866 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 23:28:14.081986 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 23:28:14.083244 18523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:14.207463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18401
I20260926 23:28:14.226807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18050
I20260926 23:28:14.227921 18111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:14.345638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18050
2026-09-26T23:28:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2968 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:14 dist-test-slave-q5g8 krb5kdc[18579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:14 dist-test-slave-q5g8 krb5kdc[18579](info): set up 2 sockets
Sep 26 23:28:14 dist-test-slave-q5g8 krb5kdc[18579](info): commencing operation
krb5kdc: starting...
W20260926 23:28:15.477061 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:15 dist-test-slave-q5g8 krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:28:15.752405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:16.128000 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:16.128412 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:16.128582 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:16.136767 18592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:16.136907 18592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:16.137024 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:16.137143 18592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:16.137240 18592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:16.147881 18592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:16.149430 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:16.150884 18592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:16.169255 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:16.169318 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:16.169262 18598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:16.170449 18592 server_base.cc:1034] running on GCE node
I20260926 23:28:16.171429 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:16.173199 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:16.174439 18592 hybrid_clock.cc:648] HybridClock initialized: now 1790465296174389 us; error 43 us; skew 500 ppm
I20260926 23:28:16.174849 18592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:16 dist-test-slave-q5g8 krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:16.178336 18592 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:28:16.180037 18592 webserver.cc:460] Webserver started at http://127.0.97.62:33447/ using document root <none> and password file <none>
I20260926 23:28:16.180877 18592 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:16.181031 18592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:16.181370 18592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:16.184577 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "8feeedd632aa4c73995b8a2768ba61fd"
format_stamp: "Formatted at 2026-09-26 23:28:16 on dist-test-slave-q5g8"
server_key: "b0dd4f8d8c1e9c2ffc0538c4a7b9bdfb"
server_key_iv: "aa18e4aa19a921e03c390513cf9eb5a1"
server_key_version: "encryptionkey@0"
I20260926 23:28:16.185540 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "8feeedd632aa4c73995b8a2768ba61fd"
format_stamp: "Formatted at 2026-09-26 23:28:16 on dist-test-slave-q5g8"
server_key: "b0dd4f8d8c1e9c2ffc0538c4a7b9bdfb"
server_key_iv: "aa18e4aa19a921e03c390513cf9eb5a1"
server_key_version: "encryptionkey@0"
I20260926 23:28:16.191313 18592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:28:16.195642 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:16.197357 18592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:28:16.197618 18592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "8feeedd632aa4c73995b8a2768ba61fd"
format_stamp: "Formatted at 2026-09-26 23:28:16 on dist-test-slave-q5g8"
server_key: "b0dd4f8d8c1e9c2ffc0538c4a7b9bdfb"
server_key_iv: "aa18e4aa19a921e03c390513cf9eb5a1"
server_key_version: "encryptionkey@0"
I20260926 23:28:16.197901 18592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:16.234953 18592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:16.240016 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:16.240383 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:16.261548 18592 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45367
I20260926 23:28:16.261610 18658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45367 every 8 connection(s)
I20260926 23:28:16.263170 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:16.267458 18659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:16.271494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18592
I20260926 23:28:16.271725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:16.272233 388 external_mini_cluster.cc:1428] Setting key 9af765a7a634b605d62f12ee8d9397d1
I20260926 23:28:16.281826 18659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd: Bootstrap starting.
I20260926 23:28:16.286188 18659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:16.287710 18659 log.cc:824] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:28:16 dist-test-slave-q5g8 krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:16.291213 18659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd: No bootstrap required, opened a new log
I20260926 23:28:16.299012 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:16.274463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.97.62:45367)
0926 23:28:16.275339 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:16.275352 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:16.275396 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:28:16.277243 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:16.277271 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:16.277757 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:16.278130 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:16.280128 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:16.281642 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:16.283687 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:16.284436 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:16.284581 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:16.290055 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:16.290065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:16.290087 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:16.290145 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:16.293853 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:16.294811 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:16.294820 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:16.294828 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:16.295036 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:16.295518 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:16.295523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:16.295527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:16.296234 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:16.296621 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:16.297249 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 23:28:16.297784 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":413,"threads_started":1}
I20260926 23:28:16.300267 18659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8feeedd632aa4c73995b8a2768ba61fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260926 23:28:16.300742 18659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:16.300886 18659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feeedd632aa4c73995b8a2768ba61fd, State: Initialized, Role: FOLLOWER
I20260926 23:28:16.301545 18659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8feeedd632aa4c73995b8a2768ba61fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260926 23:28:16.301846 18659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:16.302042 18659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:16.302246 18659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:16.304903 18659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8feeedd632aa4c73995b8a2768ba61fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260926 23:28:16.305454 18659 leader_election.cc:302] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8feeedd632aa4c73995b8a2768ba61fd; no voters:
I20260926 23:28:16.306183 18659 leader_election.cc:288] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:16.306356 18664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:16.307150 18664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [term 1 LEADER]: Becoming Leader. State: Replica: 8feeedd632aa4c73995b8a2768ba61fd, State: Running, Role: LEADER
I20260926 23:28:16.307792 18664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8feeedd632aa4c73995b8a2768ba61fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260926 23:28:16.308532 18659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:16.311043 18665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8feeedd632aa4c73995b8a2768ba61fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8feeedd632aa4c73995b8a2768ba61fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260926 23:28:16.311457 18665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:16.312059 18666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8feeedd632aa4c73995b8a2768ba61fd. Latest consensus state: current_term: 1 leader_uuid: "8feeedd632aa4c73995b8a2768ba61fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8feeedd632aa4c73995b8a2768ba61fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260926 23:28:16.312458 18666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:16.313603 18672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:16.320196 18672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:16.333966 18672 catalog_manager.cc:1347] Generated new cluster ID: 2e714a622fdb45c599ecec4c9516a3c9
I20260926 23:28:16.334116 18672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:16.349575 18672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:16.351755 18672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:16.360833 18672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd: Generated new TSK 0
I20260926 23:28:16.362077 18672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:28:16.429217 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:16.801584 18685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:16.802028 18685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:16.802150 18685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:16.810629 18685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:16.810768 18685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:16.810901 18685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:28:16.822274 18685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:16.823755 18685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:16.825278 18685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:16.840613 18693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:16.840611 18691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:16.841034 18690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:16.841120 18685 server_base.cc:1034] running on GCE node
I20260926 23:28:16.842177 18685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:16.843392 18685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:16.844754 18685 hybrid_clock.cc:648] HybridClock initialized: now 1790465296844608 us; error 83 us; skew 500 ppm
I20260926 23:28:16.845146 18685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:16 dist-test-slave-q5g8 krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:16.854432 18685 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:28:16.856187 18685 webserver.cc:460] Webserver started at http://127.0.97.1:37049/ using document root <none> and password file <none>
I20260926 23:28:16.857043 18685 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:16.857161 18685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:16.857439 18685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:16.860672 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "ad91841c0a7242a0b9836d2020ae6bd8"
format_stamp: "Formatted at 2026-09-26 23:28:16 on dist-test-slave-q5g8"
server_key: "2030beff451a39f6c5461108d58040e4"
server_key_iv: "5382f728c8bf889b33e6cd8d897bf032"
server_key_version: "encryptionkey@0"
I20260926 23:28:16.861634 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ad91841c0a7242a0b9836d2020ae6bd8"
format_stamp: "Formatted at 2026-09-26 23:28:16 on dist-test-slave-q5g8"
server_key: "2030beff451a39f6c5461108d58040e4"
server_key_iv: "5382f728c8bf889b33e6cd8d897bf032"
server_key_version: "encryptionkey@0"
I20260926 23:28:16.867175 18685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:28:16.870993 18700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:16.872608 18685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:28:16.872875 18685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "ad91841c0a7242a0b9836d2020ae6bd8"
format_stamp: "Formatted at 2026-09-26 23:28:16 on dist-test-slave-q5g8"
server_key: "2030beff451a39f6c5461108d58040e4"
server_key_iv: "5382f728c8bf889b33e6cd8d897bf032"
server_key_version: "encryptionkey@0"
I20260926 23:28:16.873114 18685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:16.894613 18685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:16.899732 18685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:16.900100 18685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:16.901319 18685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:16.903103 18685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:16.903193 18685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:16.903424 18685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:16.903506 18685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:16.940711 18685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44803
I20260926 23:28:16.940827 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44803 every 8 connection(s)
I20260926 23:28:16.942287 18685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:28:16.948645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18685
I20260926 23:28:16.948872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:28:16.949461 388 external_mini_cluster.cc:1428] Setting key 0a1a94d56f3013dcef6c3b22ffaa6ace
Sep 26 23:28:16 dist-test-slave-q5g8 krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:16.967687 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:16.945299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45841 (local address 127.0.97.62:45367)
0926 23:28:16.945855 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:16.945860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:16.946784 (+ 924us) server_negotiation.cc:407] Connection header received
0926 23:28:16.949166 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:16.949171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:16.949249 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:16.949343 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:16.951423 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:16.952224 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:16.954953 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:16.955367 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:16.955479 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:16.960322 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:16.960332 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:16.960341 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:16.960397 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:16.963344 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:16.964569 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:16.964575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:16.964578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:16.964786 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:16.965468 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:16.965486 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:16.965488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:16.965850 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:16.965993 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:16.967141 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0926 23:28:16.967376 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":314,"threads_started":1}
I20260926 23:28:16.969516 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:16.945789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.1:45841)
0926 23:28:16.946601 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:16.946642 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:16.948776 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:16.949677 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:16.949690 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:16.950498 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:16.951214 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:16.951244 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:16.952395 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:16.952403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:16.954768 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:16.954789 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:16.955643 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:16.955652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:16.956003 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:16.956852 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:16.956898 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:16.959963 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:16.963550 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:16.963563 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:16.963599 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:16.964400 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:16.964921 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:16.964929 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:16.964936 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:16.965283 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:16.966016 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:16.966028 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:16.966588 (+ 560us) client_negotiation.cc:769] Sending connection context
0926 23:28:16.966962 (+ 374us) client_negotiation.cc:240] Negotiation successful
0926 23:28:16.967682 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":341,"threads_started":1}
I20260926 23:28:16.971624 18813 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45367
I20260926 23:28:16.972014 18813 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:16.973112 18813 heartbeater.cc:511] Master 127.0.97.62:45367 requested a full tablet report, sending...
I20260926 23:28:16.975706 18624 ts_manager.cc:194] Registered new tserver with Master: ad91841c0a7242a0b9836d2020ae6bd8 (127.0.97.1:44803)
I20260926 23:28:16.977844 18624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45841
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:28:17.017390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:17.378763 18819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:17.379104 18819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:17.379276 18819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:17.388209 18819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:17.388357 18819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:17.388605 18819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:28:17.398960 18819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:17.400429 18819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:17.401921 18819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:17.416308 18827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:17.416412 18825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:17.416435 18824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:17.417029 18819 server_base.cc:1034] running on GCE node
I20260926 23:28:17.417809 18819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:17.418848 18819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:17.420095 18819 hybrid_clock.cc:648] HybridClock initialized: now 1790465297420020 us; error 68 us; skew 500 ppm
I20260926 23:28:17.420454 18819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:17 dist-test-slave-q5g8 krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:17.424015 18819 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:28:17.425753 18819 webserver.cc:460] Webserver started at http://127.0.97.2:34007/ using document root <none> and password file <none>
I20260926 23:28:17.426518 18819 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:17.426682 18819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:17.427018 18819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:17.430332 18819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "631c442903d94d55b8242a3fb7acac3a"
format_stamp: "Formatted at 2026-09-26 23:28:17 on dist-test-slave-q5g8"
server_key: "745df360834e54c990b0dc2ef35595ab"
server_key_iv: "86368b442f2ad19d9cad9394a716fc0c"
server_key_version: "encryptionkey@0"
I20260926 23:28:17.431242 18819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "631c442903d94d55b8242a3fb7acac3a"
format_stamp: "Formatted at 2026-09-26 23:28:17 on dist-test-slave-q5g8"
server_key: "745df360834e54c990b0dc2ef35595ab"
server_key_iv: "86368b442f2ad19d9cad9394a716fc0c"
server_key_version: "encryptionkey@0"
I20260926 23:28:17.437544 18819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:28:17.441690 18834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:17.443650 18819 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:28:17.443871 18819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "631c442903d94d55b8242a3fb7acac3a"
format_stamp: "Formatted at 2026-09-26 23:28:17 on dist-test-slave-q5g8"
server_key: "745df360834e54c990b0dc2ef35595ab"
server_key_iv: "86368b442f2ad19d9cad9394a716fc0c"
server_key_version: "encryptionkey@0"
I20260926 23:28:17.444135 18819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:17.466972 18819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:17.472232 18819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:17.472561 18819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:17.473685 18819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:17.475289 18819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:17.475401 18819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:17.475559 18819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:17.475634 18819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:17.512581 18819 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33995
I20260926 23:28:17.512636 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33995 every 8 connection(s)
I20260926 23:28:17.514164 18819 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:28:17.515208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18819
I20260926 23:28:17.515398 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:28:17.515884 388 external_mini_cluster.cc:1428] Setting key 5e77d94aa9647ee3ba9af604d97fbf81
Sep 26 23:28:17 dist-test-slave-q5g8 krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:17.540709 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:17.517219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55257 (local address 127.0.97.62:45367)
0926 23:28:17.517694 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:17.517700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:17.519162 (+ 1462us) server_negotiation.cc:407] Connection header received
0926 23:28:17.521325 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:17.521333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:17.521438 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:17.521573 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:17.523469 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:17.524751 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:17.526887 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:17.527406 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:17.527524 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:17.532233 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:17.532244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:17.532252 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:17.532312 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:17.536368 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:17.537516 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:17.537525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:17.537532 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:17.537748 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:17.538302 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:17.538311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:17.538317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:17.538772 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:17.538957 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:17.539847 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 23:28:17.540165 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260926 23:28:17.541863 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:17.518022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.2:55257)
0926 23:28:17.518945 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:17.519004 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:17.520986 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:17.521833 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:17.521843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:17.522555 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:17.523232 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:17.523254 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:17.524944 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:17.524948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:17.526648 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:17.526689 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:17.527715 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:17.527722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:17.528024 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:17.528880 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:17.528913 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:17.531890 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:17.536640 (+ 4750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:17.536650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:17.536669 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:17.537344 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:17.537891 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:17.537895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:17.537898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:17.538151 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:17.538936 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:17.538943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:17.539370 (+ 427us) client_negotiation.cc:769] Sending connection context
0926 23:28:17.539669 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 23:28:17.540263 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":370,"threads_started":1}
I20260926 23:28:17.543959 18947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45367
I20260926 23:28:17.544414 18947 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:28:17.545519 18947 heartbeater.cc:511] Master 127.0.97.62:45367 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:28:17.547861 18624 ts_manager.cc:194] Registered new tserver with Master: 631c442903d94d55b8242a3fb7acac3a (127.0.97.2:33995)
I20260926 23:28:17.548977 18624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:28:17.584208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:17.952227 18953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:17.952541 18953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:17.952775 18953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:17.960593 18953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:17.960808 18953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:17.961004 18953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:28:17.971390 18953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:17.972851 18953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:17.974233 18953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:28:17.981792 18813 heartbeater.cc:503] Master 127.0.97.62:45367 was elected leader, sending a full tablet report...
W20260926 23:28:17.989423 18959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:17.989562 18958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:17.989285 18961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:17.990561 18953 server_base.cc:1034] running on GCE node
I20260926 23:28:17.991387 18953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:17.992565 18953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:17.993819 18953 hybrid_clock.cc:648] HybridClock initialized: now 1790465297993771 us; error 42 us; skew 500 ppm
I20260926 23:28:17.994225 18953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:17 dist-test-slave-q5g8 krb5kdc[18579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:17.997929 18953 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:28:17.999569 18953 webserver.cc:460] Webserver started at http://127.0.97.3:39595/ using document root <none> and password file <none>
I20260926 23:28:18.000334 18953 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:18.000488 18953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:18.000895 18953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:18.004545 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "624aaf9647d54d8fac9ac9b54e54e62f"
format_stamp: "Formatted at 2026-09-26 23:28:17 on dist-test-slave-q5g8"
server_key: "d52329ca5d29a82cda1c6b53d20f7164"
server_key_iv: "b08da722edd987294fd4c748614b56be"
server_key_version: "encryptionkey@0"
I20260926 23:28:18.005471 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "624aaf9647d54d8fac9ac9b54e54e62f"
format_stamp: "Formatted at 2026-09-26 23:28:17 on dist-test-slave-q5g8"
server_key: "d52329ca5d29a82cda1c6b53d20f7164"
server_key_iv: "b08da722edd987294fd4c748614b56be"
server_key_version: "encryptionkey@0"
I20260926 23:28:18.011387 18953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:28:18.015801 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:18.017557 18953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 23:28:18.017787 18953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "624aaf9647d54d8fac9ac9b54e54e62f"
format_stamp: "Formatted at 2026-09-26 23:28:17 on dist-test-slave-q5g8"
server_key: "d52329ca5d29a82cda1c6b53d20f7164"
server_key_iv: "b08da722edd987294fd4c748614b56be"
server_key_version: "encryptionkey@0"
I20260926 23:28:18.018054 18953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:18.036702 18953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:18.042140 18953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:18.042521 18953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:18.043634 18953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:18.045295 18953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:18.045411 18953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:18.045591 18953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:18.045676 18953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:18.083431 18953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40629
I20260926 23:28:18.083483 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40629 every 8 connection(s)
I20260926 23:28:18.084964 18953 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:28:18.093140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 18953
I20260926 23:28:18.093358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:28:18.093827 388 external_mini_cluster.cc:1428] Setting key ff0903e077038206f0364179f8255b4e
Sep 26 23:28:18 dist-test-slave-q5g8 krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:18.109133 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.088571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46409 (local address 127.0.97.62:45367)
0926 23:28:18.089319 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.089324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.090172 (+ 848us) server_negotiation.cc:407] Connection header received
0926 23:28:18.092258 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.092263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.092370 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.092496 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:18.094607 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.095859 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.098099 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.098537 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.098649 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.102475 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:18.102481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:18.102484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:18.102524 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:18.105566 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.106446 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.106451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.106453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.106591 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.107017 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.107021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.107023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.107320 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:18.107443 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.108398 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.108657 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":368,"threads_started":1}
I20260926 23:28:18.110729 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.089094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.97.3:46409)
0926 23:28:18.089972 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:18.090021 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:18.091916 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:18.092854 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:18.092865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:18.093599 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:18.094351 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.094380 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.096081 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.096087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.097934 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.097951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.098773 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.098777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.098999 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.099700 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:18.099730 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:18.102188 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:18.105753 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:18.105762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:18.105780 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:18.106305 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:18.106696 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:18.106700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:18.106702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:18.106905 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:18.107436 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:18.107443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:18.107852 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 23:28:18.108204 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 23:28:18.108975 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":320,"threads_started":1}
I20260926 23:28:18.112403 19081 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45367
I20260926 23:28:18.112815 19081 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:18.113652 19081 heartbeater.cc:511] Master 127.0.97.62:45367 requested a full tablet report, sending...
I20260926 23:28:18.115468 18624 ts_manager.cc:194] Registered new tserver with Master: 624aaf9647d54d8fac9ac9b54e54e62f (127.0.97.3:40629)
I20260926 23:28:18.116318 18624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46409
I20260926 23:28:18.123459 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:28:18.143424 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.128121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.0.97.62:45367)
0926 23:28:18.128420 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.128429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.128752 (+ 323us) server_negotiation.cc:407] Connection header received
0926 23:28:18.128988 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.128994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.129083 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.129213 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:18.131094 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.131895 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.133535 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.133978 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.134134 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.137295 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:18.137300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:18.137304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:18.137347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:18.140055 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.141090 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.141094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.141096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.141239 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.141808 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.141818 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.141820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.142158 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:18.142481 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.142877 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.143098 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 23:28:18.147955 18624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:28:18.155298 18624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:28:18.184120 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.169684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.0.97.2:33995)
0926 23:28:18.170260 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.170273 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.170307 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:28:18.170742 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.170751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.171148 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.171415 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:18.172735 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.175109 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.179372 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.182377 (+ 3005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.182511 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.182731 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.183536 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.183735 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":1416,"threads_started":1}
I20260926 23:28:18.184403 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.169228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33995 (local address 127.0.0.1:58164)
0926 23:28:18.170153 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:18.170199 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:18.170408 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:18.171786 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:18.171799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:18.171845 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:18.172439 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.172457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.175373 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.175382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.179048 (+ 3666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.179081 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.182881 (+ 3800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.182890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.183170 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.183248 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 23:28:18.183422 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 23:28:18.183977 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":612,"threads_started":1}
I20260926 23:28:18.189393 18882 tablet_service.cc:1467] Processing CreateTablet for tablet ac81b6cdae6740d88b90f1877a5de374 (DEFAULT_TABLE table=test-table [id=93cd6813ca1e4fba81d8db1d5bee2074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:18.191027 18882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac81b6cdae6740d88b90f1877a5de374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:18.192937 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.170179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44803 (local address 127.0.0.1:49072)
0926 23:28:18.180223 (+ 10044us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:18.180254 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:18.180444 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:18.184741 (+ 4297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:18.184747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:18.184783 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:18.185178 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.185191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.186915 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.186921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.189439 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.189459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.191722 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.191731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.191952 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.192003 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 23:28:18.192120 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 23:28:18.192326 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9836,"mutex_wait_us":37,"thread_start_us":10749,"threads_started":1}
I20260926 23:28:18.193029 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.170944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40629 (local address 127.0.0.1:41748)
0926 23:28:18.181221 (+ 10277us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:18.181241 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:18.181350 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:18.183257 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:18.183262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:18.183291 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:18.183708 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.183720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.186838 (+ 3118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.186846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.189130 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.189151 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.191867 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.191876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.192060 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.192123 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 23:28:18.192232 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 23:28:18.192433 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10149,"mutex_wait_us":74,"thread_start_us":9608,"threads_started":1}
I20260926 23:28:18.193871 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.171302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.0.97.3:40629)
0926 23:28:18.182232 (+ 10930us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.182242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.182282 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:28:18.182392 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.182401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.182733 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.182969 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:18.185063 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.186674 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.189387 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.191166 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.191349 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.191669 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.193215 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.193426 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10714,"thread_start_us":10526,"threads_started":1}
I20260926 23:28:18.194453 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.170708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.0.97.1:44803)
0926 23:28:18.183365 (+ 12657us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.183404 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.183436 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:28:18.183548 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.183556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.183814 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.184040 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:18.185506 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.186707 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.189658 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.191417 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.191585 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.191862 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.192245 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.192475 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12475,"thread_start_us":12323,"threads_started":1}
I20260926 23:28:18.198307 18748 tablet_service.cc:1467] Processing CreateTablet for tablet ac81b6cdae6740d88b90f1877a5de374 (DEFAULT_TABLE table=test-table [id=93cd6813ca1e4fba81d8db1d5bee2074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:18.198632 19016 tablet_service.cc:1467] Processing CreateTablet for tablet ac81b6cdae6740d88b90f1877a5de374 (DEFAULT_TABLE table=test-table [id=93cd6813ca1e4fba81d8db1d5bee2074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:18.200156 18748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac81b6cdae6740d88b90f1877a5de374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:18.200456 19016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac81b6cdae6740d88b90f1877a5de374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:18.208742 19100 tablet_bootstrap.cc:492] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Bootstrap starting.
I20260926 23:28:18.213698 19100 tablet_bootstrap.cc:654] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:18.215598 19100 log.cc:824] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:18.215972 19101 tablet_bootstrap.cc:492] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Bootstrap starting.
I20260926 23:28:18.218128 19102 tablet_bootstrap.cc:492] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Bootstrap starting.
I20260926 23:28:18.220639 19101 tablet_bootstrap.cc:654] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:18.222137 19101 log.cc:824] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:18.222297 19102 tablet_bootstrap.cc:654] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:18.222693 19100 tablet_bootstrap.cc:492] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: No bootstrap required, opened a new log
I20260926 23:28:18.223022 19100 ts_tablet_manager.cc:1397] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20260926 23:28:18.223691 19102 log.cc:824] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:18.227734 19102 tablet_bootstrap.cc:492] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: No bootstrap required, opened a new log
I20260926 23:28:18.228060 19102 ts_tablet_manager.cc:1397] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 23:28:18.234470 19100 raft_consensus.cc:348] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.235021 19100 raft_consensus.cc:374] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:18.235131 19100 raft_consensus.cc:729] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631c442903d94d55b8242a3fb7acac3a, State: Initialized, Role: FOLLOWER
I20260926 23:28:18.235847 19100 consensus_queue.cc:260] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.236126 19101 tablet_bootstrap.cc:492] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: No bootstrap required, opened a new log
I20260926 23:28:18.236487 19101 ts_tablet_manager.cc:1397] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.015s
I20260926 23:28:18.237800 18947 heartbeater.cc:503] Master 127.0.97.62:45367 was elected leader, sending a full tablet report...
I20260926 23:28:18.238569 19100 ts_tablet_manager.cc:1428] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 23:28:18.240589 19102 raft_consensus.cc:348] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.241139 19102 raft_consensus.cc:374] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:18.241245 19102 raft_consensus.cc:729] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 624aaf9647d54d8fac9ac9b54e54e62f, State: Initialized, Role: FOLLOWER
I20260926 23:28:18.241981 19102 consensus_queue.cc:260] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.243332 19081 heartbeater.cc:503] Master 127.0.97.62:45367 was elected leader, sending a full tablet report...
I20260926 23:28:18.243958 19102 ts_tablet_manager.cc:1428] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260926 23:28:18.248054 19101 raft_consensus.cc:348] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.248610 19101 raft_consensus.cc:374] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:18.248740 19101 raft_consensus.cc:729] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad91841c0a7242a0b9836d2020ae6bd8, State: Initialized, Role: FOLLOWER
I20260926 23:28:18.249495 19101 consensus_queue.cc:260] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.251519 19101 ts_tablet_manager.cc:1428] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260926 23:28:18.266558 18948 tablet.cc:2369] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:28:18.337034 19082 tablet.cc:2369] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:28:18.445322 18814 tablet.cc:2369] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:18.669109 19108 raft_consensus.cc:482] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:28:18.669472 19108 raft_consensus.cc:504] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.671324 19108 leader_election.cc:288] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 631c442903d94d55b8242a3fb7acac3a (127.0.97.2:33995), 624aaf9647d54d8fac9ac9b54e54e62f (127.0.97.3:40629)
I20260926 23:28:18.678592 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.671819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58093 (local address 127.0.97.2:33995)
0926 23:28:18.672019 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.672024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.672522 (+ 498us) server_negotiation.cc:407] Connection header received
0926 23:28:18.672827 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.672833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.672923 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.673031 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:18.673685 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.674463 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.676440 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.677494 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.677681 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.677799 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.678137 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.678284 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:28:18.678728 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.671668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33995 (local address 127.0.97.1:58093)
0926 23:28:18.672394 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:18.672419 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:18.672595 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:18.673109 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:18.673114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:18.673150 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:18.673516 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.673526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.674617 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.674626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.676230 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.676245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.677763 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.677769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.677942 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.677969 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 23:28:18.678094 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 23:28:18.678277 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":1,"thread_start_us":391,"threads_started":1}
I20260926 23:28:18.679483 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.672777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49941 (local address 127.0.97.3:40629)
0926 23:28:18.672981 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.672989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.673409 (+ 420us) server_negotiation.cc:407] Connection header received
0926 23:28:18.673638 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.673642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.673707 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.673810 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:18.674376 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.675113 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.677132 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.678143 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.678294 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.678431 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.679128 (+ 697us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.679286 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:28:18.679477 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.672557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40629 (local address 127.0.97.1:49941)
0926 23:28:18.673282 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:18.673298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:18.673473 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:18.673840 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:18.673844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:18.673870 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:18.674212 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.674221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.675277 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.675289 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.676959 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:18.676971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.678538 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.678543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:18.678737 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.678785 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 23:28:18.678938 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 23:28:18.679076 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":1,"thread_start_us":384,"threads_started":1}
I20260926 23:28:18.679906 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac81b6cdae6740d88b90f1877a5de374" candidate_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631c442903d94d55b8242a3fb7acac3a" is_pre_election: true
I20260926 23:28:18.680586 18902 raft_consensus.cc:2454] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad91841c0a7242a0b9836d2020ae6bd8 in term 0.
I20260926 23:28:18.680619 19036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac81b6cdae6740d88b90f1877a5de374" candidate_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" is_pre_election: true
I20260926 23:28:18.681143 19036 raft_consensus.cc:2454] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad91841c0a7242a0b9836d2020ae6bd8 in term 0.
I20260926 23:28:18.681427 18703 leader_election.cc:302] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631c442903d94d55b8242a3fb7acac3a, ad91841c0a7242a0b9836d2020ae6bd8; no voters:
I20260926 23:28:18.681883 19108 raft_consensus.cc:2781] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:28:18.682050 19108 raft_consensus.cc:482] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:28:18.682168 19108 raft_consensus.cc:3037] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:18.684491 19108 raft_consensus.cc:504] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.685216 19108 leader_election.cc:288] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 631c442903d94d55b8242a3fb7acac3a (127.0.97.2:33995), 624aaf9647d54d8fac9ac9b54e54e62f (127.0.97.3:40629)
I20260926 23:28:18.685781 18902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac81b6cdae6740d88b90f1877a5de374" candidate_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631c442903d94d55b8242a3fb7acac3a"
I20260926 23:28:18.685887 19036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac81b6cdae6740d88b90f1877a5de374" candidate_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624aaf9647d54d8fac9ac9b54e54e62f"
I20260926 23:28:18.686110 18902 raft_consensus.cc:3037] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:18.686163 19036 raft_consensus.cc:3037] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:18.688494 18902 raft_consensus.cc:2454] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad91841c0a7242a0b9836d2020ae6bd8 in term 1.
I20260926 23:28:18.688498 19036 raft_consensus.cc:2454] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad91841c0a7242a0b9836d2020ae6bd8 in term 1.
I20260926 23:28:18.689030 18703 leader_election.cc:302] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631c442903d94d55b8242a3fb7acac3a, ad91841c0a7242a0b9836d2020ae6bd8; no voters:
I20260926 23:28:18.689400 19108 raft_consensus.cc:2781] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:18.689898 19108 raft_consensus.cc:686] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 1 LEADER]: Becoming Leader. State: Replica: ad91841c0a7242a0b9836d2020ae6bd8, State: Running, Role: LEADER
I20260926 23:28:18.690472 19108 consensus_queue.cc:237] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } }
I20260926 23:28:18.695245 18624 catalog_manager.cc:5495] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad91841c0a7242a0b9836d2020ae6bd8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad91841c0a7242a0b9836d2020ae6bd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44803 } health_report { overall_health: HEALTHY } } }
I20260926 23:28:18.765749 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.759471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.97.1:44803)
0926 23:28:18.759955 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.759960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.760079 (+ 119us) server_negotiation.cc:407] Connection header received
0926 23:28:18.760273 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.760277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.760355 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.760497 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:28:18.761052 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.761779 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.763241 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.763650 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.763756 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.764496 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:28:18.765120 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:28:18.765261 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.765462 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.765506 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":285,"threads_started":1}
I20260926 23:28:18.774446 18902 raft_consensus.cc:1261] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 1 FOLLOWER]: Refusing update from remote peer ad91841c0a7242a0b9836d2020ae6bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:18.774447 19036 raft_consensus.cc:1261] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 1 FOLLOWER]: Refusing update from remote peer ad91841c0a7242a0b9836d2020ae6bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:18.775535 19111 consensus_queue.cc:1035] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "631c442903d94d55b8242a3fb7acac3a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:18.775908 19108 consensus_queue.cc:1035] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "624aaf9647d54d8fac9ac9b54e54e62f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:18.804966 18624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:18.805316 18624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:18.808864 18624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8feeedd632aa4c73995b8a2768ba61fd: Sending DeleteTablet for 3 replicas of tablet ac81b6cdae6740d88b90f1877a5de374
I20260926 23:28:18.810171 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet ac81b6cdae6740d88b90f1877a5de374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58164
I20260926 23:28:18.810544 18748 tablet_service.cc:1514] Processing DeleteTablet for tablet ac81b6cdae6740d88b90f1877a5de374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49072
I20260926 23:28:18.810921 19016 tablet_service.cc:1514] Processing DeleteTablet for tablet ac81b6cdae6740d88b90f1877a5de374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41748
I20260926 23:28:18.811585 19132 tablet_replica.cc:330] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: stopping tablet replica
I20260926 23:28:18.811841 19133 tablet_replica.cc:330] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: stopping tablet replica
I20260926 23:28:18.812053 19132 raft_consensus.cc:2229] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:28:18.812269 19133 raft_consensus.cc:2229] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:18.812762 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18685
I20260926 23:28:18.812790 19133 raft_consensus.cc:2258] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:18.813351 19132 raft_consensus.cc:2258] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:18.814879 19133 ts_tablet_manager.cc:1905] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:18.815263 19132 ts_tablet_manager.cc:1905] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:18.822507 19133 ts_tablet_manager.cc:1918] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:18.822781 19133 log.cc:1196] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/ac81b6cdae6740d88b90f1877a5de374
I20260926 23:28:18.823391 19133 ts_tablet_manager.cc:1939] T ac81b6cdae6740d88b90f1877a5de374 P 624aaf9647d54d8fac9ac9b54e54e62f: Deleting consensus metadata
I20260926 23:28:18.824673 19132 ts_tablet_manager.cc:1918] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:18.824910 19132 log.cc:1196] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/ac81b6cdae6740d88b90f1877a5de374
I20260926 23:28:18.825759 18610 catalog_manager.cc:4841] TS 624aaf9647d54d8fac9ac9b54e54e62f (127.0.97.3:40629): tablet ac81b6cdae6740d88b90f1877a5de374 (table test-table [id=93cd6813ca1e4fba81d8db1d5bee2074]) successfully deleted
I20260926 23:28:18.826416 19132 ts_tablet_manager.cc:1939] T ac81b6cdae6740d88b90f1877a5de374 P ad91841c0a7242a0b9836d2020ae6bd8: Deleting consensus metadata
I20260926 23:28:18.827489 19131 tablet_replica.cc:330] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: stopping tablet replica
I20260926 23:28:18.828017 19131 raft_consensus.cc:2229] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:18.828682 19131 raft_consensus.cc:2258] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:18.828871 18611 catalog_manager.cc:4841] TS ad91841c0a7242a0b9836d2020ae6bd8 (127.0.97.1:44803): tablet ac81b6cdae6740d88b90f1877a5de374 (table test-table [id=93cd6813ca1e4fba81d8db1d5bee2074]) successfully deleted
I20260926 23:28:18.830775 19131 ts_tablet_manager.cc:1905] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:28:18 dist-test-slave-q5g8 krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:28:18.839470 19131 ts_tablet_manager.cc:1918] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:18.839705 19131 log.cc:1196] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/ac81b6cdae6740d88b90f1877a5de374
I20260926 23:28:18.840325 19131 ts_tablet_manager.cc:1939] T ac81b6cdae6740d88b90f1877a5de374 P 631c442903d94d55b8242a3fb7acac3a: Deleting consensus metadata
I20260926 23:28:18.842448 18610 catalog_manager.cc:4841] TS 631c442903d94d55b8242a3fb7acac3a (127.0.97.2:33995): tablet ac81b6cdae6740d88b90f1877a5de374 (table test-table [id=93cd6813ca1e4fba81d8db1d5bee2074]) successfully deleted
I20260926 23:28:18.849220 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.815239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.0.97.1:44803)
0926 23:28:18.815424 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.815430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.826731 (+ 11301us) server_negotiation.cc:407] Connection header received
0926 23:28:18.826974 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.826983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.827108 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.827235 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:18.829753 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.831139 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.833315 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.834089 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.834257 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.840775 (+ 6518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:18.840798 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:18.840820 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:18.840880 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:18.845559 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.846437 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.846448 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.846452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.846647 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.847046 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.847051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.847055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.847550 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:18.847894 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.848363 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.848722 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:28:18.849993 18808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:18.957553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18685
I20260926 23:28:18.977651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18819
Sep 26 23:28:18 dist-test-slave-q5g8 krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:28:18.996812 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:18.978636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.0.97.2:33995)
0926 23:28:18.978848 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:18.978853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:18.978877 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:28:18.979078 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:18.979083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:18.979173 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:18.979269 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:18.980794 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.981650 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.983151 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:18.983658 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:18.983851 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:18.989225 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:18.989245 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:18.989265 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:18.989314 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:18.992949 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.993979 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.993990 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.993993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.994182 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:18.994714 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:18.994721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:18.994725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:18.995257 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:18.995627 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:18.996014 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 23:28:18.996286 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:28:18.997709 18942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:19.097963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18819
I20260926 23:28:19.117561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18953
Sep 26 23:28:19 dist-test-slave-q5g8 krb5kdc[18579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:28:19.134498 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:19.118630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.97.3:40629)
0926 23:28:19.118861 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:19.118866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:19.118889 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:28:19.118951 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:19.118955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:19.119039 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:19.119149 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:19.120652 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:19.121640 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:19.123145 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:19.123554 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:19.123669 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:19.128266 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:19.128284 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:19.128301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:19.128347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:19.131575 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:19.132393 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:19.132401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:19.132403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:19.132565 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:19.133091 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:19.133095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:19.133097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:19.133514 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:19.133778 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:19.134038 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 23:28:19.134220 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 23:28:19.135325 19076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:19.230571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18953
I20260926 23:28:19.249068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 18592
I20260926 23:28:19.250409 18654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:19.372892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 18592
2026-09-26T23:28:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5029 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T23:28:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:19Z Disabled control of system clock
I20260926 23:28:19.453311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43513
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:28:19.833244 19142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:19.833632 19142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:19.833729 19142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:19.842036 19142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:19.842187 19142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:19.842228 19142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:19.842280 19142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:19.842327 19142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:19.852540 19142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:19.854164 19142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:19.856101 19142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:19.869503 19148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:19.869629 19147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:19.870121 19150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:19.871443 19142 server_base.cc:1034] running on GCE node
I20260926 23:28:19.872406 19142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:19.874075 19142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:19.875305 19142 hybrid_clock.cc:648] HybridClock initialized: now 1790465299875254 us; error 41 us; skew 500 ppm
I20260926 23:28:19.875741 19142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:19.878382 19142 webserver.cc:460] Webserver started at http://127.0.97.62:41469/ using document root <none> and password file <none>
I20260926 23:28:19.879168 19142 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:19.879295 19142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:19.879653 19142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:19.882944 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "2c4e6d4f7d2b46068cada4ca61b75d01"
format_stamp: "Formatted at 2026-09-26 23:28:19 on dist-test-slave-q5g8"
server_key: "0bc91f032a5d6ff3df7b9b895e57eb4d"
server_key_iv: "d68148baa0d0a62723796dc4f8eb2730"
server_key_version: "encryptionkey@0"
I20260926 23:28:19.883836 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c4e6d4f7d2b46068cada4ca61b75d01"
format_stamp: "Formatted at 2026-09-26 23:28:19 on dist-test-slave-q5g8"
server_key: "0bc91f032a5d6ff3df7b9b895e57eb4d"
server_key_iv: "d68148baa0d0a62723796dc4f8eb2730"
server_key_version: "encryptionkey@0"
I20260926 23:28:19.889797 19142 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260926 23:28:19.894107 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:19.895671 19142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:28:19.895910 19142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "2c4e6d4f7d2b46068cada4ca61b75d01"
format_stamp: "Formatted at 2026-09-26 23:28:19 on dist-test-slave-q5g8"
server_key: "0bc91f032a5d6ff3df7b9b895e57eb4d"
server_key_iv: "d68148baa0d0a62723796dc4f8eb2730"
server_key_version: "encryptionkey@0"
I20260926 23:28:19.896265 19142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:19.911533 19142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:19.912523 19142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:19.912927 19142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:19.933235 19142 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43513
I20260926 23:28:19.933256 19207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43513 every 8 connection(s)
I20260926 23:28:19.934774 19142 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:19.935240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 19142
I20260926 23:28:19.935410 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:19.935933 388 external_mini_cluster.cc:1428] Setting key 21e33529007745d9f551b1a3747dc167
I20260926 23:28:19.939687 19208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:19.951431 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:19.938143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.0.97.62:43513)
0926 23:28:19.939223 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:19.939241 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:19.939290 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:28:19.941184 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:19.941208 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:19.941723 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:19.942107 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:19.942788 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:19.944138 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:19.946268 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:19.947017 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:19.947172 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:19.948635 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:19.948645 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:19.948661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:19.949348 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:19.949438 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:19.949465 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:19.949574 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:19.949863 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 23:28:19.950275 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":404,"threads_started":1}
I20260926 23:28:19.953675 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01: Bootstrap starting.
I20260926 23:28:19.957324 19208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:19.958384 19208 log.cc:824] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:19.961686 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01: No bootstrap required, opened a new log
I20260926 23:28:19.969556 19208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43513 } }
I20260926 23:28:19.970037 19208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:19.970150 19208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4e6d4f7d2b46068cada4ca61b75d01, State: Initialized, Role: FOLLOWER
I20260926 23:28:19.970827 19208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43513 } }
I20260926 23:28:19.971100 19208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:19.971251 19208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:19.971434 19208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:19.974056 19208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43513 } }
I20260926 23:28:19.974678 19208 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4e6d4f7d2b46068cada4ca61b75d01; no voters:
I20260926 23:28:19.975374 19208 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:19.975576 19213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:19.976470 19213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4e6d4f7d2b46068cada4ca61b75d01, State: Running, Role: LEADER
I20260926 23:28:19.977242 19213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43513 } }
I20260926 23:28:19.977883 19208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:19.980762 19214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43513 } } }
I20260926 23:28:19.980841 19215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4e6d4f7d2b46068cada4ca61b75d01. Latest consensus state: current_term: 1 leader_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4e6d4f7d2b46068cada4ca61b75d01" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43513 } } }
I20260926 23:28:19.981277 19214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:19.981344 19215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:19.982436 19220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:19.989630 19220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:20.000519 19220 catalog_manager.cc:1347] Generated new cluster ID: 1dfbd64baf7d44ddbf5e647a92c4ce12
I20260926 23:28:20.000705 19220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:20.037887 19220 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:20.040014 19220 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:20.051481 19220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01: Generated new TSK 0
I20260926 23:28:20.052829 19220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:20.063357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43513
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:28:20.445178 19232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:20.445506 19232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:20.445600 19232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:20.453274 19232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:20.453399 19232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:20.453512 19232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:28:20.463557 19232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:20.465199 19232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:20.467088 19232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:20.481710 19238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:20.481966 19237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:20.482076 19240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:20.482852 19232 server_base.cc:1034] running on GCE node
I20260926 23:28:20.483741 19232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:20.485114 19232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:20.486404 19232 hybrid_clock.cc:648] HybridClock initialized: now 1790465300486346 us; error 54 us; skew 500 ppm
I20260926 23:28:20.486832 19232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:20.489851 19232 webserver.cc:460] Webserver started at http://127.0.97.1:37697/ using document root <none> and password file <none>
I20260926 23:28:20.490753 19232 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:20.490924 19232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:20.491328 19232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:20.494923 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "d480af3bed704a25815adc3d0ad6a1e3"
format_stamp: "Formatted at 2026-09-26 23:28:20 on dist-test-slave-q5g8"
server_key: "48e0fd228d55a0a65e6a28232b94a044"
server_key_iv: "9395588da42159b008107c78b6f47004"
server_key_version: "encryptionkey@0"
I20260926 23:28:20.495956 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d480af3bed704a25815adc3d0ad6a1e3"
format_stamp: "Formatted at 2026-09-26 23:28:20 on dist-test-slave-q5g8"
server_key: "48e0fd228d55a0a65e6a28232b94a044"
server_key_iv: "9395588da42159b008107c78b6f47004"
server_key_version: "encryptionkey@0"
I20260926 23:28:20.502673 19232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260926 23:28:20.507172 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:20.509025 19232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:28:20.509262 19232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "d480af3bed704a25815adc3d0ad6a1e3"
format_stamp: "Formatted at 2026-09-26 23:28:20 on dist-test-slave-q5g8"
server_key: "48e0fd228d55a0a65e6a28232b94a044"
server_key_iv: "9395588da42159b008107c78b6f47004"
server_key_version: "encryptionkey@0"
I20260926 23:28:20.509567 19232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:20.528836 19232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:20.529767 19232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:20.530074 19232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:20.531152 19232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:20.532825 19232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:20.532964 19232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:20.533105 19232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:20.533196 19232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:20.570752 19232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44645
I20260926 23:28:20.570814 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44645 every 8 connection(s)
I20260926 23:28:20.572314 19232 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:28:20.573400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 19232
I20260926 23:28:20.573585 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:28:20.574128 388 external_mini_cluster.cc:1428] Setting key 62cad708a77f8a8c7440020901be8a6e
I20260926 23:28:20.577826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43513
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:28:20.591589 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:20.575668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54439 (local address 127.0.97.62:43513)
0926 23:28:20.576204 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:20.576212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:20.578107 (+ 1895us) server_negotiation.cc:407] Connection header received
0926 23:28:20.581339 (+ 3232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:20.581344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:20.581436 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:20.581557 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:20.583355 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:20.584487 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:20.587668 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:20.588230 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:20.588398 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:20.590533 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:20.590539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:20.590542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:20.590568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:20.590616 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:20.590631 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:20.590728 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:20.591170 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 23:28:20.591279 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":266,"threads_started":1}
I20260926 23:28:20.593286 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:20.576502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43513 (local address 127.0.97.1:54439)
0926 23:28:20.577823 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:20.577875 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:20.580927 (+ 3052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:20.581928 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:20.581945 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:20.582101 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:20.583106 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:20.583141 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:20.584770 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:20.584778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:20.587369 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:20.587396 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:20.588574 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:20.588584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:20.589040 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:20.590005 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:20.590012 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:20.590078 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:20.590221 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:20.590758 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:20.590774 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:20.590787 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 23:28:20.591006 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 23:28:20.591553 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":396,"threads_started":1}
I20260926 23:28:20.595412 19359 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43513
I20260926 23:28:20.595813 19359 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:20.596860 19359 heartbeater.cc:511] Master 127.0.97.62:43513 requested a full tablet report, sending...
I20260926 23:28:20.599650 19173 ts_manager.cc:194] Registered new tserver with Master: d480af3bed704a25815adc3d0ad6a1e3 (127.0.97.1:44645)
I20260926 23:28:20.602341 19173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54439
W20260926 23:28:20.943048 19363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:20.943360 19363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:20.943519 19363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:20.951713 19363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:20.951835 19363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:20.952052 19363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:28:20.961812 19363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:20.963323 19363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:20.965492 19363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:20.980280 19369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:20.980307 19368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:20.980562 19371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:20.981346 19363 server_base.cc:1034] running on GCE node
I20260926 23:28:20.982065 19363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:20.983284 19363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:20.984572 19363 hybrid_clock.cc:648] HybridClock initialized: now 1790465300984498 us; error 52 us; skew 500 ppm
I20260926 23:28:20.985013 19363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:20.987794 19363 webserver.cc:460] Webserver started at http://127.0.97.2:33047/ using document root <none> and password file <none>
I20260926 23:28:20.988653 19363 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:20.988876 19363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:20.989280 19363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:20.992791 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a"
format_stamp: "Formatted at 2026-09-26 23:28:20 on dist-test-slave-q5g8"
server_key: "80db5733910dadbdf7e67b267af15e98"
server_key_iv: "c0b03f4d2333189a6d04b6067db76e5f"
server_key_version: "encryptionkey@0"
I20260926 23:28:20.993804 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a"
format_stamp: "Formatted at 2026-09-26 23:28:20 on dist-test-slave-q5g8"
server_key: "80db5733910dadbdf7e67b267af15e98"
server_key_iv: "c0b03f4d2333189a6d04b6067db76e5f"
server_key_version: "encryptionkey@0"
I20260926 23:28:21.000289 19363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:28:21.005031 19377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:21.006839 19363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:28:21.007094 19363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a"
format_stamp: "Formatted at 2026-09-26 23:28:20 on dist-test-slave-q5g8"
server_key: "80db5733910dadbdf7e67b267af15e98"
server_key_iv: "c0b03f4d2333189a6d04b6067db76e5f"
server_key_version: "encryptionkey@0"
I20260926 23:28:21.007395 19363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:21.043880 19363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:21.044992 19363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:21.045332 19363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:21.046586 19363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:21.048455 19363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:21.048631 19363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:21.048820 19363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:21.048929 19363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:21.090920 19363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42247
I20260926 23:28:21.091034 19489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42247 every 8 connection(s)
I20260926 23:28:21.092638 19363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:28:21.102985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 19363
I20260926 23:28:21.103157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:28:21.103648 388 external_mini_cluster.cc:1428] Setting key aaf17d19bb278797ddcc510c50db74b2
I20260926 23:28:21.106812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43513
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:28:21.108006 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.095986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37415 (local address 127.0.97.62:43513)
0926 23:28:21.096550 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.096555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.098081 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 23:28:21.100107 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.100117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.100198 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.100367 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:21.101797 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.102585 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.104507 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.105098 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.105209 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.106914 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:21.106920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:21.106925 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:21.106953 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:21.107015 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:21.107037 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:21.107137 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.107399 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.107524 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":293,"threads_started":1}
I20260926 23:28:21.108949 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.096820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43513 (local address 127.0.97.2:37415)
0926 23:28:21.097888 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:21.097931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:21.099762 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:21.100655 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:21.100665 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:21.100847 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:21.101564 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.101608 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.102765 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.102769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.104336 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.104353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.105338 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.105345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.105705 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.106549 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:21.106552 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:21.106595 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:21.106686 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:21.107171 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:21.107180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:21.107187 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:28:21.107348 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 23:28:21.107683 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":336,"threads_started":1}
I20260926 23:28:21.110545 19490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43513
I20260926 23:28:21.110932 19490 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:21.112089 19490 heartbeater.cc:511] Master 127.0.97.62:43513 requested a full tablet report, sending...
I20260926 23:28:21.113932 19173 ts_manager.cc:194] Registered new tserver with Master: 8c02bd2ecbc04ae2bfe6581e01d27b4a (127.0.97.2:42247)
I20260926 23:28:21.114969 19173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37415
W20260926 23:28:21.478875 19494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:21.479357 19494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:21.479525 19494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:21.487749 19494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:21.487941 19494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:21.488147 19494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:28:21.499338 19494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:21.500967 19494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:21.503149 19494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:21.518630 19499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:21.518939 19500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:21.519788 19502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:21.520283 19494 server_base.cc:1034] running on GCE node
I20260926 23:28:21.521060 19494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:21.522190 19494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:21.523475 19494 hybrid_clock.cc:648] HybridClock initialized: now 1790465301523406 us; error 63 us; skew 500 ppm
I20260926 23:28:21.523854 19494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:21.526541 19494 webserver.cc:460] Webserver started at http://127.0.97.3:35181/ using document root <none> and password file <none>
I20260926 23:28:21.527374 19494 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:21.527523 19494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:21.527884 19494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:21.531219 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "94084e6db0674becba2a73581a21068f"
format_stamp: "Formatted at 2026-09-26 23:28:21 on dist-test-slave-q5g8"
server_key: "cef2908df3db9da8ce56a4ac97deb19a"
server_key_iv: "79e5e4e0bdaef7405c313351a869c775"
server_key_version: "encryptionkey@0"
I20260926 23:28:21.532212 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "94084e6db0674becba2a73581a21068f"
format_stamp: "Formatted at 2026-09-26 23:28:21 on dist-test-slave-q5g8"
server_key: "cef2908df3db9da8ce56a4ac97deb19a"
server_key_iv: "79e5e4e0bdaef7405c313351a869c775"
server_key_version: "encryptionkey@0"
I20260926 23:28:21.538612 19494 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:28:21.543159 19508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:21.545079 19494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:28:21.545317 19494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "94084e6db0674becba2a73581a21068f"
format_stamp: "Formatted at 2026-09-26 23:28:21 on dist-test-slave-q5g8"
server_key: "cef2908df3db9da8ce56a4ac97deb19a"
server_key_iv: "79e5e4e0bdaef7405c313351a869c775"
server_key_version: "encryptionkey@0"
I20260926 23:28:21.545598 19494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:21.559270 19494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:21.560290 19494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:21.560626 19494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:21.561821 19494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:21.563627 19494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:21.563766 19494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:21.563923 19494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:21.564028 19494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:21.603365 19494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36143
I20260926 23:28:21.603431 19620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36143 every 8 connection(s)
I20260926 23:28:21.605130 19494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:28:21.606545 19359 heartbeater.cc:503] Master 127.0.97.62:43513 was elected leader, sending a full tablet report...
I20260926 23:28:21.607846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 19494
I20260926 23:28:21.608031 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:28:21.608639 388 external_mini_cluster.cc:1428] Setting key e4d8baa7d9f1b782e47c8e86bdf49bb0
I20260926 23:28:21.620128 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.608712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38855 (local address 127.0.97.62:43513)
0926 23:28:21.609609 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.609622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.610962 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 23:28:21.613148 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.613153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.613229 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.613324 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:21.614552 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.615463 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.617257 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.617686 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.617794 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.619334 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:21.619339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:21.619342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:21.619363 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:21.619406 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:21.619418 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:21.619493 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.619734 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.619825 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":626,"threads_started":1}
I20260926 23:28:21.621466 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.609480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43513 (local address 127.0.97.3:38855)
0926 23:28:21.610782 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:21.610842 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:21.612855 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:21.613530 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:21.613541 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:21.613649 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:21.614387 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.614406 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.615638 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.615642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.617106 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.617120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.617900 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.617905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.618157 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.618874 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:21.618876 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:21.618922 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:21.619010 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:21.619520 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:21.619529 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:21.619536 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:28:21.619706 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 23:28:21.620098 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":421,"threads_started":1}
I20260926 23:28:21.623260 19621 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43513
I20260926 23:28:21.623620 19621 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:21.624459 19621 heartbeater.cc:511] Master 127.0.97.62:43513 requested a full tablet report, sending...
I20260926 23:28:21.626171 19173 ts_manager.cc:194] Registered new tserver with Master: 94084e6db0674becba2a73581a21068f (127.0.97.3:36143)
I20260926 23:28:21.627108 19173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:38855
I20260926 23:28:21.637564 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:28:21.649191 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.642416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.0.97.62:43513)
0926 23:28:21.642689 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.642698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.643007 (+ 309us) server_negotiation.cc:407] Connection header received
0926 23:28:21.643244 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.643250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.643327 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.643432 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:21.643945 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.644792 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.646342 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.646772 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.646878 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.648203 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:21.648208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:21.648212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:21.648232 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:21.648268 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:21.648280 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:21.648410 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.648655 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.648794 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:28:21.653985 19173 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:28:21.661742 19173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:28:21.706923 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.676017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42247 (local address 127.0.0.1:33860)
0926 23:28:21.677089 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:21.677156 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:21.677413 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:21.697316 (+ 19903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:21.697330 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:21.697389 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:21.698015 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.698034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.700055 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.700064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.703055 (+ 2991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.703143 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.705689 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.705697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.705970 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.706046 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 23:28:21.706211 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 23:28:21.706474 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":41,"thread_start_us":862,"threads_started":1}
I20260926 23:28:21.707345 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.678560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36143 (local address 127.0.0.1:47260)
0926 23:28:21.697777 (+ 19217us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:21.697798 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:21.698067 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:21.698845 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:21.698851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:21.698880 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:21.699237 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.699249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.701385 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.701393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.704028 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.704043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.706452 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.706458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.706669 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.706705 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:28:21.706800 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 23:28:21.706970 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19062,"mutex_wait_us":1,"thread_start_us":16748,"threads_started":1}
I20260926 23:28:21.708256 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.676237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.0.97.2:42247)
0926 23:28:21.696306 (+ 20069us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.696320 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.696368 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:28:21.696519 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.696532 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.696896 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.697792 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:21.698321 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.699909 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.703681 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.705434 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.705602 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.705934 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.707603 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.707823 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19810,"thread_start_us":19638,"threads_started":1}
I20260926 23:28:21.708287 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.679042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.0.97.3:36143)
0926 23:28:21.688801 (+ 9759us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.688816 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.697931 (+ 9115us) server_negotiation.cc:407] Connection header received
0926 23:28:21.698304 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.698319 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.698588 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.698840 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:21.699455 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.701176 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.704318 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.705814 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.705983 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.706307 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.707663 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.707837 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9419,"thread_start_us":9205,"threads_started":1}
I20260926 23:28:21.713318 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.676613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.0.97.1:44645)
0926 23:28:21.694814 (+ 18201us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.694829 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.696216 (+ 1387us) server_negotiation.cc:407] Connection header received
0926 23:28:21.696365 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.696375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.696778 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.697021 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:21.698615 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.700318 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.703488 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.710744 (+ 7256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.710919 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.711233 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.712641 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.712884 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17902,"thread_start_us":18192,"threads_started":1}
I20260926 23:28:21.712425 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.676272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48602)
0926 23:28:21.695773 (+ 19501us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:21.695803 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:21.696017 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:21.697789 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:21.697797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:21.697840 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:21.698404 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.698420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.700512 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.700518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.703098 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.703120 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.711476 (+ 8356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.711488 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.711697 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.711756 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 23:28:21.711870 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:28:21.712025 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19298,"mutex_wait_us":92,"thread_start_us":21140,"threads_started":1}
I20260926 23:28:21.713102 19556 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee (DEFAULT_TABLE table=test-table [id=294cad3a2b74467a8dfdee27c88bbacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:21.713063 19425 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee (DEFAULT_TABLE table=test-table [id=294cad3a2b74467a8dfdee27c88bbacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:21.714850 19556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:21.714854 19425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:21.718081 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee (DEFAULT_TABLE table=test-table [id=294cad3a2b74467a8dfdee27c88bbacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:21.719779 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:21.728356 19640 tablet_bootstrap.cc:492] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Bootstrap starting.
I20260926 23:28:21.731184 19641 tablet_bootstrap.cc:492] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Bootstrap starting.
I20260926 23:28:21.733690 19640 tablet_bootstrap.cc:654] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:21.735256 19640 log.cc:824] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:21.735898 19642 tablet_bootstrap.cc:492] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: Bootstrap starting.
I20260926 23:28:21.736562 19641 tablet_bootstrap.cc:654] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:21.738715 19641 log.cc:824] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:21.739745 19642 tablet_bootstrap.cc:654] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:21.741456 19642 log.cc:824] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:21.743629 19641 tablet_bootstrap.cc:492] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: No bootstrap required, opened a new log
I20260926 23:28:21.743908 19640 tablet_bootstrap.cc:492] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: No bootstrap required, opened a new log
I20260926 23:28:21.744047 19641 ts_tablet_manager.cc:1397] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 23:28:21.744244 19640 ts_tablet_manager.cc:1397] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260926 23:28:21.755667 19642 tablet_bootstrap.cc:492] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: No bootstrap required, opened a new log
I20260926 23:28:21.756040 19642 ts_tablet_manager.cc:1397] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20260926 23:28:21.756844 19640 raft_consensus.cc:348] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.757407 19640 raft_consensus.cc:374] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:21.757165 19641 raft_consensus.cc:348] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.757562 19640 raft_consensus.cc:729] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c02bd2ecbc04ae2bfe6581e01d27b4a, State: Initialized, Role: FOLLOWER
I20260926 23:28:21.757658 19641 raft_consensus.cc:374] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:21.757779 19641 raft_consensus.cc:729] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94084e6db0674becba2a73581a21068f, State: Initialized, Role: FOLLOWER
I20260926 23:28:21.758426 19641 consensus_queue.cc:260] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.758421 19640 consensus_queue.cc:260] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.759784 19490 heartbeater.cc:503] Master 127.0.97.62:43513 was elected leader, sending a full tablet report...
I20260926 23:28:21.760236 19621 heartbeater.cc:503] Master 127.0.97.62:43513 was elected leader, sending a full tablet report...
I20260926 23:28:21.762480 19640 ts_tablet_manager.cc:1428] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260926 23:28:21.762470 19641 ts_tablet_manager.cc:1428] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Time spent starting tablet: real 0.018s user 0.006s sys 0.011s
I20260926 23:28:21.766089 19642 raft_consensus.cc:348] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.766628 19642 raft_consensus.cc:374] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:21.766778 19642 raft_consensus.cc:729] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d480af3bed704a25815adc3d0ad6a1e3, State: Initialized, Role: FOLLOWER
I20260926 23:28:21.767655 19642 consensus_queue.cc:260] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.769958 19642 ts_tablet_manager.cc:1428] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260926 23:28:21.775436 19648 raft_consensus.cc:482] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:28:21.775789 19648 raft_consensus.cc:504] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.777832 19648 leader_election.cc:288] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c02bd2ecbc04ae2bfe6581e01d27b4a (127.0.97.2:42247), 94084e6db0674becba2a73581a21068f (127.0.97.3:36143)
I20260926 23:28:21.786273 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.778216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47289 (local address 127.0.97.2:42247)
0926 23:28:21.778458 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.778467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.779651 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 23:28:21.779876 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.779880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.779958 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.780070 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:21.780842 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.781623 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.784160 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.785153 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.785262 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.785392 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.785824 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.785965 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:28:21.786286 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.778545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43909 (local address 127.0.97.3:36143)
0926 23:28:21.778809 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.778819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.779519 (+ 700us) server_negotiation.cc:407] Connection header received
0926 23:28:21.779746 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.779750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.779813 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.779903 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:21.780639 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.781436 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.783816 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.785188 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.785368 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.785477 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.785881 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.786032 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:28:21.786401 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.778228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42247 (local address 127.0.97.1:47289)
0926 23:28:21.779534 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:21.779550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:21.779721 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:21.780089 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:21.780094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:21.780137 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:21.780619 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.780629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.781791 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.781797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.783906 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.783921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.785371 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.785378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.785640 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.785679 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 23:28:21.785765 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:28:21.785947 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"mutex_wait_us":39,"thread_start_us":942,"threads_started":1}
I20260926 23:28:21.786435 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.778207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36143 (local address 127.0.97.1:43909)
0926 23:28:21.779395 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:21.779412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:21.779598 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:21.779932 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:21.779937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:21.779960 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:21.780477 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.780488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.781582 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.781589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.783556 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:21.783570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.785436 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.785444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:21.785681 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.785707 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 23:28:21.785824 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 23:28:21.785969 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":195,"spinlock_wait_cycles":33664,"thread_start_us":635,"threads_started":1}
I20260926 23:28:21.788003 19576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2c9ee0861414799e6ac4aaefdb5ee" candidate_uuid: "d480af3bed704a25815adc3d0ad6a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94084e6db0674becba2a73581a21068f" is_pre_election: true
I20260926 23:28:21.788275 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2c9ee0861414799e6ac4aaefdb5ee" candidate_uuid: "d480af3bed704a25815adc3d0ad6a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" is_pre_election: true
I20260926 23:28:21.788631 19576 raft_consensus.cc:2454] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d480af3bed704a25815adc3d0ad6a1e3 in term 0.
I20260926 23:28:21.788972 19445 raft_consensus.cc:2454] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d480af3bed704a25815adc3d0ad6a1e3 in term 0.
I20260926 23:28:21.789556 19248 leader_election.cc:302] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94084e6db0674becba2a73581a21068f, d480af3bed704a25815adc3d0ad6a1e3; no voters:
I20260926 23:28:21.790066 19648 raft_consensus.cc:2781] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:28:21.790256 19648 raft_consensus.cc:482] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:28:21.790381 19648 raft_consensus.cc:3037] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:21.792860 19648 raft_consensus.cc:504] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.793557 19648 leader_election.cc:288] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [CANDIDATE]: Term 1 election: Requested vote from peers 8c02bd2ecbc04ae2bfe6581e01d27b4a (127.0.97.2:42247), 94084e6db0674becba2a73581a21068f (127.0.97.3:36143)
I20260926 23:28:21.794152 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2c9ee0861414799e6ac4aaefdb5ee" candidate_uuid: "d480af3bed704a25815adc3d0ad6a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a"
I20260926 23:28:21.794169 19576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb2c9ee0861414799e6ac4aaefdb5ee" candidate_uuid: "d480af3bed704a25815adc3d0ad6a1e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94084e6db0674becba2a73581a21068f"
I20260926 23:28:21.794370 19445 raft_consensus.cc:3037] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:21.794389 19576 raft_consensus.cc:3037] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:21.796818 19576 raft_consensus.cc:2454] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d480af3bed704a25815adc3d0ad6a1e3 in term 1.
I20260926 23:28:21.796836 19445 raft_consensus.cc:2454] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d480af3bed704a25815adc3d0ad6a1e3 in term 1.
I20260926 23:28:21.797353 19248 leader_election.cc:302] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8c02bd2ecbc04ae2bfe6581e01d27b4a, 94084e6db0674becba2a73581a21068f, d480af3bed704a25815adc3d0ad6a1e3; no voters:
I20260926 23:28:21.797770 19648 raft_consensus.cc:2781] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:21.798282 19648 raft_consensus.cc:686] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 1 LEADER]: Becoming Leader. State: Replica: d480af3bed704a25815adc3d0ad6a1e3, State: Running, Role: LEADER
I20260926 23:28:21.798858 19648 consensus_queue.cc:237] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } }
I20260926 23:28:21.804016 19172 catalog_manager.cc:5495] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d480af3bed704a25815adc3d0ad6a1e3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d480af3bed704a25815adc3d0ad6a1e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d480af3bed704a25815adc3d0ad6a1e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 } health_report { overall_health: UNKNOWN } } }
I20260926 23:28:21.822376 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.816466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.0.97.1:44645)
0926 23:28:21.816650 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.816655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.816718 (+ 63us) server_negotiation.cc:407] Connection header received
0926 23:28:21.816955 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.816960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.817062 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.817213 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:28:21.817655 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.818361 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.820041 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.820459 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.820571 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.821182 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:28:21.821735 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:28:21.821934 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.822055 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.822108 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 23:28:21.825003 19360 tablet.cc:2369] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:21.831717 19445 raft_consensus.cc:1261] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 1 FOLLOWER]: Refusing update from remote peer d480af3bed704a25815adc3d0ad6a1e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:21.831717 19576 raft_consensus.cc:1261] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 1 FOLLOWER]: Refusing update from remote peer d480af3bed704a25815adc3d0ad6a1e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:21.832962 19651 consensus_queue.cc:1035] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c02bd2ecbc04ae2bfe6581e01d27b4a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:21.833658 19648 consensus_queue.cc:1035] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94084e6db0674becba2a73581a21068f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:28:21.845283 19491 tablet.cc:2369] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:21.848227 19655 mvcc.cc:205] Tried to move back new op lower bound from 7333745876290334720 to 7333745876172611584. Current Snapshot: MvccSnapshot[applied={T|T < 7333745876290334720}]
I20260926 23:28:21.850452 19656 mvcc.cc:205] Tried to move back new op lower bound from 7333745876290334720 to 7333745876172611584. Current Snapshot: MvccSnapshot[applied={T|T < 7333745876290334720}]
I20260926 23:28:21.862457 19172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:21.862794 19172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:21.866889 19172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c4e6d4f7d2b46068cada4ca61b75d01: Sending DeleteTablet for 3 replicas of tablet 8fb2c9ee0861414799e6ac4aaefdb5ee
I20260926 23:28:21.868248 19425 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:21 UTC) from {username='slave'} at 127.0.0.1:33860
I20260926 23:28:21.868254 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:21 UTC) from {username='slave'} at 127.0.0.1:48602
I20260926 23:28:21.868556 19556 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:21 UTC) from {username='slave'} at 127.0.0.1:47260
I20260926 23:28:21.875180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 19232
I20260926 23:28:21.885207 19666 tablet_replica.cc:330] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: stopping tablet replica
I20260926 23:28:21.885958 19666 raft_consensus.cc:2229] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:21.886724 19666 raft_consensus.cc:2258] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:21.889348 19666 ts_tablet_manager.cc:1905] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:21.897176 19666 ts_tablet_manager.cc:1918] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:21.897477 19666 log.cc:1196] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/8fb2c9ee0861414799e6ac4aaefdb5ee
I20260926 23:28:21.898157 19666 ts_tablet_manager.cc:1939] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 94084e6db0674becba2a73581a21068f: Deleting consensus metadata
I20260926 23:28:21.900451 19664 tablet_replica.cc:330] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: stopping tablet replica
I20260926 23:28:21.900545 19158 catalog_manager.cc:4841] TS 94084e6db0674becba2a73581a21068f (127.0.97.3:36143): tablet 8fb2c9ee0861414799e6ac4aaefdb5ee (table test-table [id=294cad3a2b74467a8dfdee27c88bbacc]) successfully deleted
I20260926 23:28:21.901671 19664 raft_consensus.cc:2229] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:21.902415 19664 raft_consensus.cc:2258] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:21.903648 19665 tablet_replica.cc:330] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: stopping tablet replica
I20260926 23:28:21.904134 19665 raft_consensus.cc:2229] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:28:21.904933 19665 raft_consensus.cc:2258] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:21.905099 19664 ts_tablet_manager.cc:1905] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:21.907143 19665 ts_tablet_manager.cc:1905] T 8fb2c9ee0861414799e6ac4aaefdb5ee P d480af3bed704a25815adc3d0ad6a1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:21.911094 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:21.877149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.0.97.1:44645)
0926 23:28:21.877418 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:21.877425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:21.892740 (+ 15315us) server_negotiation.cc:407] Connection header received
0926 23:28:21.893395 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:21.893405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:21.893557 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:21.893691 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:21.894619 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.896361 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.900892 (+ 4531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:21.901950 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:21.902104 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:21.908964 (+ 6860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:21.908981 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:21.909004 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:21.909947 (+ 943us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:21.910025 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:21.910057 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:21.910163 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:21.910505 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 23:28:21.910657 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 23:28:21.912266 19664 ts_tablet_manager.cc:1918] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:21.912501 19664 log.cc:1196] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/8fb2c9ee0861414799e6ac4aaefdb5ee
I20260926 23:28:21.912742 19354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:21.913200 19664 ts_tablet_manager.cc:1939] T 8fb2c9ee0861414799e6ac4aaefdb5ee P 8c02bd2ecbc04ae2bfe6581e01d27b4a: Deleting consensus metadata
I20260926 23:28:21.915508 19157 catalog_manager.cc:4841] TS 8c02bd2ecbc04ae2bfe6581e01d27b4a (127.0.97.2:42247): tablet 8fb2c9ee0861414799e6ac4aaefdb5ee (table test-table [id=294cad3a2b74467a8dfdee27c88bbacc]) successfully deleted
I20260926 23:28:22.022174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 19232
W20260926 23:28:22.043712 19160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44645 (error 108)
I20260926 23:28:22.043828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 19363
I20260926 23:28:22.045997 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.045508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48636)
0926 23:28:22.045705 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.045836 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 23:28:22.046641 19160 catalog_manager.cc:4584] TS d480af3bed704a25815adc3d0ad6a1e3 (127.0.97.1:44645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fb2c9ee0861414799e6ac4aaefdb5ee: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
I20260926 23:28:22.052304 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.044826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.0.97.2:42247)
0926 23:28:22.045096 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:22.045104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:22.045136 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:28:22.045388 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:22.045393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:22.045504 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:22.045655 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:22.046084 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:22.047082 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:22.048561 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:22.049290 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:22.049446 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:22.050722 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:22.050733 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:22.050750 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:22.051419 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:22.051475 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:22.051498 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:22.051662 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:22.051802 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 23:28:22.051932 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 23:28:22.053189 19485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:22.100612 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.099948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48644)
0926 23:28:22.100262 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.100410 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 23:28:22.101962 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.101528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48650)
0926 23:28:22.101723 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.101838 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 23:28:22.154026 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 19363
I20260926 23:28:22.162766 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.162197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48666)
0926 23:28:22.162444 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.162545 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 23:28:22.163980 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.163603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48676)
0926 23:28:22.163779 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.163863 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 23:28:22.173697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 19494
I20260926 23:28:22.180902 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.174573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.0.97.3:36143)
0926 23:28:22.174757 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:22.174761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:22.174781 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:28:22.174850 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:22.174855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:22.174957 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:22.175052 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:22.175446 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:22.176223 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:22.177672 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:22.178153 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:22.178263 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:22.179412 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:22.179422 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:22.179438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:22.180084 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:22.180157 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:22.180185 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:22.180285 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:22.180413 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 23:28:22.180524 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:28:22.181720 19616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:22.244943 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.244315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48692)
0926 23:28:22.244590 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.244754 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 23:28:22.246170 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.245709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48704)
0926 23:28:22.245968 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.246045 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 23:28:22.279721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 19494
I20260926 23:28:22.297839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 19142
I20260926 23:28:22.299226 19203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:22.436297 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:22.435477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44645 (local address 127.0.0.1:48710)
0926 23:28:22.435886 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:22.436048 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
I20260926 23:28:22.436331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 19142
2026-09-26T23:28:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3060 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:22 dist-test-slave-q5g8 krb5kdc[19675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:22 dist-test-slave-q5g8 krb5kdc[19675](info): set up 2 sockets
Sep 26 23:28:22 dist-test-slave-q5g8 krb5kdc[19675](info): commencing operation
krb5kdc: starting...
W20260926 23:28:23.593103 388 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:23 dist-test-slave-q5g8 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:28:23.873307 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33845
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:24.228750 19688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:24.229090 19688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:24.229184 19688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:24.236755 19688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:24.236912 19688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:24.236970 19688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:24.237030 19688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:24.237074 19688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:24.248965 19688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:24.250483 19688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:24.251991 19688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:24.266238 19693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:24.266297 19696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:24.266324 19694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:24.267292 19688 server_base.cc:1034] running on GCE node
I20260926 23:28:24.268234 19688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:24.270156 19688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:24.271430 19688 hybrid_clock.cc:648] HybridClock initialized: now 1790465304271371 us; error 54 us; skew 500 ppm
I20260926 23:28:24.271912 19688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:24 dist-test-slave-q5g8 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:24.275866 19688 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:28:24.277856 19688 webserver.cc:460] Webserver started at http://127.0.97.62:46297/ using document root <none> and password file <none>
I20260926 23:28:24.278710 19688 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:24.278893 19688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:24.279251 19688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:24.282703 19688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "0b80674aa8fd4db08bc0c8923ac187f3"
format_stamp: "Formatted at 2026-09-26 23:28:24 on dist-test-slave-q5g8"
server_key: "31fa48ed19aee2f0f6d800d74e42fab3"
server_key_iv: "38a75faf1eec2ccc10ff17daf140f1a8"
server_key_version: "encryptionkey@0"
I20260926 23:28:24.283574 19688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "0b80674aa8fd4db08bc0c8923ac187f3"
format_stamp: "Formatted at 2026-09-26 23:28:24 on dist-test-slave-q5g8"
server_key: "31fa48ed19aee2f0f6d800d74e42fab3"
server_key_iv: "38a75faf1eec2ccc10ff17daf140f1a8"
server_key_version: "encryptionkey@0"
I20260926 23:28:24.289767 19688 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:28:24.294178 19703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:24.295783 19688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:28:24.296018 19688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "0b80674aa8fd4db08bc0c8923ac187f3"
format_stamp: "Formatted at 2026-09-26 23:28:24 on dist-test-slave-q5g8"
server_key: "31fa48ed19aee2f0f6d800d74e42fab3"
server_key_iv: "38a75faf1eec2ccc10ff17daf140f1a8"
server_key_version: "encryptionkey@0"
I20260926 23:28:24.296289 19688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:24.323464 19688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:24.328744 19688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:24.329105 19688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:24.350176 19688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33845
I20260926 23:28:24.350209 19754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33845 every 8 connection(s)
I20260926 23:28:24.351663 19688 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:24.355589 19755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:24.361861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 19688
I20260926 23:28:24.362073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:24.362645 388 external_mini_cluster.cc:1428] Setting key 1bd062c73384c8dadcf22afd6468d099
I20260926 23:28:24.367720 19755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3: Bootstrap starting.
I20260926 23:28:24.371111 19755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:24.372208 19755 log.cc:824] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:24.375874 19755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3: No bootstrap required, opened a new log
Sep 26 23:28:24 dist-test-slave-q5g8 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:24.387153 19755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33845 } }
I20260926 23:28:24.387779 19755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:24.387916 19755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b80674aa8fd4db08bc0c8923ac187f3, State: Initialized, Role: FOLLOWER
I20260926 23:28:24.388661 19755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33845 } }
I20260926 23:28:24.389024 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:24.364896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.0.97.62:33845)
0926 23:28:24.365760 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:24.365775 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:24.365818 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:28:24.367584 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:24.367612 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:24.368121 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:24.368538 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:24.370183 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:24.371604 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:24.373272 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:24.374265 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:24.374477 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:24.379099 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:24.379111 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:24.379136 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:24.379196 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:24.383625 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:24.384744 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:24.384754 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:24.384764 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:24.385017 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:24.385589 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:24.385599 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:24.385601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:24.386299 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:24.386550 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:24.387159 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 23:28:24.387631 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":383,"threads_started":1}
I20260926 23:28:24.389026 19755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:24.389299 19755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:24.389508 19755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:24.392549 19755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33845 } }
I20260926 23:28:24.393148 19755 leader_election.cc:302] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b80674aa8fd4db08bc0c8923ac187f3; no voters:
I20260926 23:28:24.393725 19755 leader_election.cc:288] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:24.393884 19760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:24.394655 19760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0b80674aa8fd4db08bc0c8923ac187f3, State: Running, Role: LEADER
I20260926 23:28:24.395217 19760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33845 } }
I20260926 23:28:24.395956 19755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:24.398110 19762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b80674aa8fd4db08bc0c8923ac187f3. Latest consensus state: current_term: 1 leader_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33845 } } }
I20260926 23:28:24.398526 19762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:24.400456 19765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:24.400900 19761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b80674aa8fd4db08bc0c8923ac187f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33845 } } }
I20260926 23:28:24.401338 19761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:24.405752 19765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:24.415477 19765 catalog_manager.cc:1347] Generated new cluster ID: 4eec90b6067c4f228397929a810447cf
I20260926 23:28:24.415632 19765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:24.462988 19765 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:24.465085 19765 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:24.476333 19765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3: Generated new TSK 0
I20260926 23:28:24.477324 19765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:28:24.552871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33845
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:24.916546 19781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:24.916963 19781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:24.917095 19781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:24.924881 19781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:24.925027 19781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:24.925153 19781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:28:24.936105 19781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:24.937711 19781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:24.939376 19781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:24.954784 19789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:24.954742 19786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:24.954814 19787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:24.955802 19781 server_base.cc:1034] running on GCE node
I20260926 23:28:24.956525 19781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:24.957657 19781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:24.958950 19781 hybrid_clock.cc:648] HybridClock initialized: now 1790465304958887 us; error 42 us; skew 500 ppm
I20260926 23:28:24.959388 19781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:24 dist-test-slave-q5g8 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:24.963147 19781 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:28:24.964996 19781 webserver.cc:460] Webserver started at http://127.0.97.1:40123/ using document root <none> and password file <none>
I20260926 23:28:24.965752 19781 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:24.965900 19781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:24.966233 19781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:24.969455 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "32e99679d3ba404a93ee74c1f52f0d67"
format_stamp: "Formatted at 2026-09-26 23:28:24 on dist-test-slave-q5g8"
server_key: "674a57fe766ca1cf2c85a906750fc25e"
server_key_iv: "935904f2761f4b5280354e7118ba8f29"
server_key_version: "encryptionkey@0"
I20260926 23:28:24.970299 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "32e99679d3ba404a93ee74c1f52f0d67"
format_stamp: "Formatted at 2026-09-26 23:28:24 on dist-test-slave-q5g8"
server_key: "674a57fe766ca1cf2c85a906750fc25e"
server_key_iv: "935904f2761f4b5280354e7118ba8f29"
server_key_version: "encryptionkey@0"
I20260926 23:28:24.975943 19781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:28:24.979982 19796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:24.981472 19781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:28:24.981678 19781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "32e99679d3ba404a93ee74c1f52f0d67"
format_stamp: "Formatted at 2026-09-26 23:28:24 on dist-test-slave-q5g8"
server_key: "674a57fe766ca1cf2c85a906750fc25e"
server_key_iv: "935904f2761f4b5280354e7118ba8f29"
server_key_version: "encryptionkey@0"
I20260926 23:28:24.981889 19781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:25.014212 19781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:25.019222 19781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:25.019601 19781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:25.020839 19781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:25.022383 19781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:25.022495 19781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:25.022653 19781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:25.022734 19781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:25.058712 19781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46223
I20260926 23:28:25.058780 19908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46223 every 8 connection(s)
I20260926 23:28:25.060268 19781 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:28:25.060752 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 19781
I20260926 23:28:25.060952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:28:25.061462 388 external_mini_cluster.cc:1428] Setting key 4d607dd45c468be506af832c5f25e874
Sep 26 23:28:25 dist-test-slave-q5g8 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:28:25.093766 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:25.063839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34333 (local address 127.0.97.62:33845)
0926 23:28:25.069249 (+ 5410us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:25.069259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:25.070051 (+ 792us) server_negotiation.cc:407] Connection header received
0926 23:28:25.072514 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:25.072520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:25.072621 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:25.072817 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:25.075598 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.076648 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.079835 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.080346 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.080487 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:25.086514 (+ 6027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:25.086525 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:25.086530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:25.086587 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:25.089782 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:25.091047 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:25.091052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:25.091055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:25.091201 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:25.091856 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:25.091860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:25.091862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:25.092173 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:25.092291 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:25.093257 (+ 966us) server_negotiation.cc:299] Negotiation successful
0926 23:28:25.093477 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5226,"thread_start_us":5012,"threads_started":1}
I20260926 23:28:25.094888 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:25.064899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33845 (local address 127.0.97.1:34333)
0926 23:28:25.069792 (+ 4893us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:25.069861 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:25.072042 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:25.073184 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:25.073205 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:25.074211 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:25.075207 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:25.075240 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.076922 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.076932 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:25.079577 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:25.079604 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.080716 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.080729 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:25.081133 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:25.082123 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:25.082166 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:25.086048 (+ 3882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:25.090015 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:25.090031 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:25.090057 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:25.090856 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:25.091368 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:25.091376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:25.091381 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:25.091719 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:25.092337 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:25.092344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:25.092810 (+ 466us) client_negotiation.cc:769] Sending connection context
0926 23:28:25.093132 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 23:28:25.093605 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4517,"thread_start_us":4210,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:28:25.096895 19909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33845
I20260926 23:28:25.097265 19909 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:25.098091 19909 heartbeater.cc:511] Master 127.0.97.62:33845 requested a full tablet report, sending...
I20260926 23:28:25.100615 19720 ts_manager.cc:194] Registered new tserver with Master: 32e99679d3ba404a93ee74c1f52f0d67 (127.0.97.1:46223)
I20260926 23:28:25.102528 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:28:25.129123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33845
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:25.484081 19915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:25.484390 19915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:25.484545 19915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:25.492856 19915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:25.493008 19915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:25.493239 19915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:28:25.505193 19915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:25.506619 19915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:25.508031 19915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:25.522346 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:25.522326 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:25.522568 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:25.523765 19915 server_base.cc:1034] running on GCE node
I20260926 23:28:25.524454 19915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:25.525738 19915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:25.527010 19915 hybrid_clock.cc:648] HybridClock initialized: now 1790465305526935 us; error 63 us; skew 500 ppm
I20260926 23:28:25.527341 19915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:25 dist-test-slave-q5g8 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:25.531139 19915 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:28:25.532945 19915 webserver.cc:460] Webserver started at http://127.0.97.2:37429/ using document root <none> and password file <none>
I20260926 23:28:25.533787 19915 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:25.533967 19915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:25.534339 19915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:25.537873 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "51b26ca20d754ce5822512efaf778e6f"
format_stamp: "Formatted at 2026-09-26 23:28:25 on dist-test-slave-q5g8"
server_key: "84db268c89c8ea7fc9c6d1d5403a8cfd"
server_key_iv: "915b0a5050dd4bcc93d70a66781e2e56"
server_key_version: "encryptionkey@0"
I20260926 23:28:25.538833 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "51b26ca20d754ce5822512efaf778e6f"
format_stamp: "Formatted at 2026-09-26 23:28:25 on dist-test-slave-q5g8"
server_key: "84db268c89c8ea7fc9c6d1d5403a8cfd"
server_key_iv: "915b0a5050dd4bcc93d70a66781e2e56"
server_key_version: "encryptionkey@0"
I20260926 23:28:25.545012 19915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 23:28:25.549320 19930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:25.551086 19915 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:28:25.551311 19915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "51b26ca20d754ce5822512efaf778e6f"
format_stamp: "Formatted at 2026-09-26 23:28:25 on dist-test-slave-q5g8"
server_key: "84db268c89c8ea7fc9c6d1d5403a8cfd"
server_key_iv: "915b0a5050dd4bcc93d70a66781e2e56"
server_key_version: "encryptionkey@0"
I20260926 23:28:25.551579 19915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:25.588567 19915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:25.594563 19915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:25.594851 19915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:25.596086 19915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:25.597822 19915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:25.597911 19915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:25.598019 19915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:25.598080 19915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:25.636878 19915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41745
I20260926 23:28:25.636940 20042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41745 every 8 connection(s)
I20260926 23:28:25.638504 19915 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:28:25.647806 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 19915
I20260926 23:28:25.648018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:28:25.648646 388 external_mini_cluster.cc:1428] Setting key aef10ca6a3e2c055e3ecfbff6a10a6d7
Sep 26 23:28:25 dist-test-slave-q5g8 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:25.664215 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:25.641409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33641 (local address 127.0.97.62:33845)
0926 23:28:25.641878 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:25.641883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:25.643081 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 23:28:25.645243 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:25.645249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:25.645356 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:25.645498 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:25.647259 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.648439 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.650495 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.651218 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.651353 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:25.656170 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:25.656180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:25.656189 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:25.656244 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:25.660294 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:25.661433 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:25.661438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:25.661441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:25.661611 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:25.662152 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:25.662156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:25.662159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:25.662499 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:25.662691 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:25.663511 (+ 820us) server_negotiation.cc:299] Negotiation successful
0926 23:28:25.663781 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20260926 23:28:25.665432 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:25.642041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33845 (local address 127.0.97.2:33641)
0926 23:28:25.642855 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:25.642909 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:25.644923 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:25.645783 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:25.645793 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:25.646408 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:25.647044 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:25.647064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.648634 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.648638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:25.650321 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:25.650336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:25.651497 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:25.651505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:25.651853 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:25.652737 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:25.652778 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:25.655852 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:25.660527 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:25.660537 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:25.660554 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:25.661240 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:25.661763 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:25.661768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:25.661772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:25.661999 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:25.662658 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:25.662667 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:25.663086 (+ 419us) client_negotiation.cc:769] Sending connection context
0926 23:28:25.663406 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 23:28:25.664046 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":334,"threads_started":1}
I20260926 23:28:25.667255 20043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33845
I20260926 23:28:25.667632 20043 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:25.668509 20043 heartbeater.cc:511] Master 127.0.97.62:33845 requested a full tablet report, sending...
I20260926 23:28:25.670387 19720 ts_manager.cc:194] Registered new tserver with Master: 51b26ca20d754ce5822512efaf778e6f (127.0.97.2:41745)
I20260926 23:28:25.671794 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33641
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:28:25.713479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33845
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:26.072191 20049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:26.072543 20049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:26.072649 20049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:26.080790 20049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:26.080950 20049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:26.081086 20049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:28:26.092278 20049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:26.093789 20049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:26.095199 20049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:28:26.106981 19909 heartbeater.cc:503] Master 127.0.97.62:33845 was elected leader, sending a full tablet report...
W20260926 23:28:26.111249 20057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:26.111204 20054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:26.111738 20055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:26.112026 20049 server_base.cc:1034] running on GCE node
I20260926 23:28:26.112921 20049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:26.114144 20049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:26.115437 20049 hybrid_clock.cc:648] HybridClock initialized: now 1790465306115377 us; error 62 us; skew 500 ppm
I20260926 23:28:26.115864 20049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:26 dist-test-slave-q5g8 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:26.119902 20049 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:28:26.121742 20049 webserver.cc:460] Webserver started at http://127.0.97.3:43925/ using document root <none> and password file <none>
I20260926 23:28:26.122494 20049 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:26.122614 20049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:26.122901 20049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:26.126243 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "3993e4208ef74160b16b6e96084a0698"
format_stamp: "Formatted at 2026-09-26 23:28:26 on dist-test-slave-q5g8"
server_key: "804689c5c2c66213c7600a607dfa3e34"
server_key_iv: "8d8a0d1634e3d2e61933be90133a98cc"
server_key_version: "encryptionkey@0"
I20260926 23:28:26.127053 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3993e4208ef74160b16b6e96084a0698"
format_stamp: "Formatted at 2026-09-26 23:28:26 on dist-test-slave-q5g8"
server_key: "804689c5c2c66213c7600a607dfa3e34"
server_key_iv: "8d8a0d1634e3d2e61933be90133a98cc"
server_key_version: "encryptionkey@0"
I20260926 23:28:26.132984 20049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:28:26.136953 20064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:26.138594 20049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:28:26.138818 20049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "3993e4208ef74160b16b6e96084a0698"
format_stamp: "Formatted at 2026-09-26 23:28:26 on dist-test-slave-q5g8"
server_key: "804689c5c2c66213c7600a607dfa3e34"
server_key_iv: "8d8a0d1634e3d2e61933be90133a98cc"
server_key_version: "encryptionkey@0"
I20260926 23:28:26.139072 20049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:26.154531 20049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:26.159703 20049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:26.159986 20049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:26.161196 20049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:26.162904 20049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:26.162995 20049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:26.163097 20049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:26.163162 20049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:26.200361 20049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33551
I20260926 23:28:26.200381 20176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33551 every 8 connection(s)
I20260926 23:28:26.202119 20049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:28:26.210487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 20049
I20260926 23:28:26.210685 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:28:26.211166 388 external_mini_cluster.cc:1428] Setting key aa6ca3efe8ec4839ed4a204a57d0141e
Sep 26 23:28:26 dist-test-slave-q5g8 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:26.224296 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.205154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35811 (local address 127.0.97.62:33845)
0926 23:28:26.205622 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.205627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.206836 (+ 1209us) server_negotiation.cc:407] Connection header received
0926 23:28:26.208992 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.208998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.209112 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.209247 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:26.211162 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.212067 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.213866 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.214320 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.214467 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.218292 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:26.218297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:26.218301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:26.218341 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:26.221348 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.222146 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.222150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.222152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.222276 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.222668 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.222671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.222673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.222938 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:26.223057 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.223800 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.223978 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260926 23:28:26.225399 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.205810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33845 (local address 127.0.97.3:35811)
0926 23:28:26.206644 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:26.206688 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:26.208579 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:26.209542 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:26.209553 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:26.210197 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:26.210961 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.210982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.212253 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.212258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.213692 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.213706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.214583 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.214591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.214850 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.215551 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:28:26.215576 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:26.218054 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:26.221533 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:26.221541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:26.221557 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:26.222026 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:26.222366 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:28:26.222370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:28:26.222372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:28:26.222568 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:28:26.223080 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:26.223086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:26.223462 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 23:28:26.223696 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 23:28:26.224129 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":346,"threads_started":1}
I20260926 23:28:26.226887 20177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33845
I20260926 23:28:26.227259 20177 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:26.227988 20177 heartbeater.cc:511] Master 127.0.97.62:33845 requested a full tablet report, sending...
I20260926 23:28:26.229530 19719 ts_manager.cc:194] Registered new tserver with Master: 3993e4208ef74160b16b6e96084a0698 (127.0.97.3:33551)
I20260926 23:28:26.230480 19719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35811
I20260926 23:28:26.238955 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:28:26.255278 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.243064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.0.97.62:33845)
0926 23:28:26.243346 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.243352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.243603 (+ 251us) server_negotiation.cc:407] Connection header received
0926 23:28:26.243746 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.243750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.243815 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.243897 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:26.245448 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.246187 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.247391 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.247774 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.247904 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.250282 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:26.250286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:26.250290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:26.250326 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:26.252796 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.253424 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.253428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.253430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.253555 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.253880 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.253883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.253885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.254212 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:26.254410 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.254719 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.254907 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 23:28:26.259541 19719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:28:26.266680 19719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:28:26.292708 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.279951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46223 (local address 127.0.0.1:38498)
0926 23:28:26.281880 (+ 1929us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:26.281933 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:26.282132 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:26.285723 (+ 3591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:26.285729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:26.285754 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:26.286116 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.286128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.288046 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.288052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.289966 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.289979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.291558 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.291565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.291785 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.291845 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 23:28:26.291969 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 23:28:26.292202 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1695,"mutex_wait_us":77,"thread_start_us":5095,"threads_started":1}
I20260926 23:28:26.293205 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.280424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.0.97.3:33551)
0926 23:28:26.283374 (+ 2950us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.283387 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.283429 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:28:26.283572 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.283584 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.283913 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.284167 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:26.285102 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.287115 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.290293 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.291737 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.291875 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.292133 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.292629 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.292840 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2740,"thread_start_us":2577,"threads_started":1}
I20260926 23:28:26.293229 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.279819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33551 (local address 127.0.0.1:47762)
0926 23:28:26.282244 (+ 2425us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:26.282266 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:26.282433 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:26.284209 (+ 1776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:26.284221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:26.284267 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:26.284850 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.284863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.287293 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.287299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.290072 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.290099 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.292291 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.292295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.292422 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.292445 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:28:26.292548 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 23:28:26.292721 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":92,"thread_start_us":2530,"threads_started":1}
I20260926 23:28:26.293907 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.281229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41745 (local address 127.0.0.1:40546)
0926 23:28:26.285370 (+ 4141us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:26.285390 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:26.285525 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:26.286162 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:26.286171 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:26.286196 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:26.286591 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.286603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.288456 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.288462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.290796 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.290809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.292998 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.293004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.293201 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.293239 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:28:26.293350 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:28:26.293493 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3997,"mutex_wait_us":57,"thread_start_us":247,"threads_started":1}
I20260926 23:28:26.294688 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.281217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.0.97.1:46223)
0926 23:28:26.285083 (+ 3866us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.285091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.285120 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:28:26.285223 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.285231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.285526 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.285729 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:26.286337 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.287898 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.290134 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.291315 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.293849 (+ 2534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.294122 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.294184 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.294346 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3652,"thread_start_us":3468,"threads_started":1}
I20260926 23:28:26.294697 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.281548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.0.97.2:41745)
0926 23:28:26.282753 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.282766 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.285574 (+ 2808us) server_negotiation.cc:407] Connection header received
0926 23:28:26.285702 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.285709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.285926 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.286157 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:26.286730 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.288294 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.291016 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.292445 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.292594 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.292843 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.294163 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.294344 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":910,"threads_started":1}
I20260926 23:28:26.297644 20112 tablet_service.cc:1467] Processing CreateTablet for tablet c2042e887117463a8d7b46bb122b7b48 (DEFAULT_TABLE table=test-table [id=fa39686979484874b91859864ce62ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:26.298338 19844 tablet_service.cc:1467] Processing CreateTablet for tablet c2042e887117463a8d7b46bb122b7b48 (DEFAULT_TABLE table=test-table [id=fa39686979484874b91859864ce62ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:26.299057 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2042e887117463a8d7b46bb122b7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:26.299010 19978 tablet_service.cc:1467] Processing CreateTablet for tablet c2042e887117463a8d7b46bb122b7b48 (DEFAULT_TABLE table=test-table [id=fa39686979484874b91859864ce62ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:26.299841 19844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2042e887117463a8d7b46bb122b7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:26.300396 19978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2042e887117463a8d7b46bb122b7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:26.313225 20196 tablet_bootstrap.cc:492] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Bootstrap starting.
I20260926 23:28:26.316865 20196 tablet_bootstrap.cc:654] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:26.318558 20196 log.cc:824] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:26.324616 20197 tablet_bootstrap.cc:492] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Bootstrap starting.
I20260926 23:28:26.325740 20198 tablet_bootstrap.cc:492] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Bootstrap starting.
I20260926 23:28:26.329669 20198 tablet_bootstrap.cc:654] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:26.330920 20198 log.cc:824] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:26.331403 20197 tablet_bootstrap.cc:654] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:26.332816 20197 log.cc:824] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:26.334156 20196 tablet_bootstrap.cc:492] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: No bootstrap required, opened a new log
I20260926 23:28:26.334293 20198 tablet_bootstrap.cc:492] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: No bootstrap required, opened a new log
I20260926 23:28:26.334497 20196 ts_tablet_manager.cc:1397] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.008s
I20260926 23:28:26.334544 20198 ts_tablet_manager.cc:1397] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260926 23:28:26.335539 20197 tablet_bootstrap.cc:492] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: No bootstrap required, opened a new log
I20260926 23:28:26.335814 20197 ts_tablet_manager.cc:1397] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 23:28:26.343487 20197 raft_consensus.cc:348] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.343984 20197 raft_consensus.cc:374] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:26.344079 20197 raft_consensus.cc:729] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e99679d3ba404a93ee74c1f52f0d67, State: Initialized, Role: FOLLOWER
I20260926 23:28:26.344851 20197 consensus_queue.cc:260] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.345707 20198 raft_consensus.cc:348] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.345747 20196 raft_consensus.cc:348] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.346177 20198 raft_consensus.cc:374] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:26.346177 20196 raft_consensus.cc:374] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:26.346269 20196 raft_consensus.cc:729] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3993e4208ef74160b16b6e96084a0698, State: Initialized, Role: FOLLOWER
I20260926 23:28:26.346273 20198 raft_consensus.cc:729] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b26ca20d754ce5822512efaf778e6f, State: Initialized, Role: FOLLOWER
I20260926 23:28:26.346917 20196 consensus_queue.cc:260] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.346912 20198 consensus_queue.cc:260] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.348876 20043 heartbeater.cc:503] Master 127.0.97.62:33845 was elected leader, sending a full tablet report...
I20260926 23:28:26.349025 20196 ts_tablet_manager.cc:1428] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260926 23:28:26.349376 20198 ts_tablet_manager.cc:1428] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260926 23:28:26.350646 20177 heartbeater.cc:503] Master 127.0.97.62:33845 was elected leader, sending a full tablet report...
I20260926 23:28:26.354382 20197 ts_tablet_manager.cc:1428] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
W20260926 23:28:26.390924 20044 tablet.cc:2369] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:26.394339 20203 raft_consensus.cc:482] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:28:26.394639 20203 raft_consensus.cc:504] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.396519 20203 leader_election.cc:288] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e99679d3ba404a93ee74c1f52f0d67 (127.0.97.1:46223), 51b26ca20d754ce5822512efaf778e6f (127.0.97.2:41745)
I20260926 23:28:26.403872 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.396978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39765 (local address 127.0.97.2:41745)
0926 23:28:26.397201 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.397208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.397690 (+ 482us) server_negotiation.cc:407] Connection header received
0926 23:28:26.397933 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.397939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.398031 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.398129 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:26.398642 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.399557 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.401989 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.402859 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.402966 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.403041 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.403459 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.403609 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:28:26.403880 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.397104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41745 (local address 127.0.97.3:39765)
0926 23:28:26.397553 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:26.397583 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:26.397762 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:26.398199 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:26.398205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:26.398239 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:26.398481 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.398496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.399702 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.399708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.401793 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.401829 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.403034 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.403043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.403275 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.403315 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 23:28:26.403426 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:28:26.403568 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260926 23:28:26.404150 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.397210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55759 (local address 127.0.97.1:46223)
0926 23:28:26.397393 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.397397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.398077 (+ 680us) server_negotiation.cc:407] Connection header received
0926 23:28:26.398255 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.398261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.398343 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.398452 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:26.399028 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.400147 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.402220 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.403162 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.403306 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.403421 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.403740 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.403882 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:28:26.404239 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.397147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46223 (local address 127.0.97.3:55759)
0926 23:28:26.397939 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:26.397957 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:26.398115 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:26.398471 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:26.398474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:26.398493 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:26.398867 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.398878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.400344 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.400349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.402054 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:26.402068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.403377 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.403398 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:26.403565 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.403598 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:28:26.403690 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:28:26.403815 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":425,"threads_started":1}
I20260926 23:28:26.405421 19998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2042e887117463a8d7b46bb122b7b48" candidate_uuid: "3993e4208ef74160b16b6e96084a0698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b26ca20d754ce5822512efaf778e6f" is_pre_election: true
I20260926 23:28:26.405699 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2042e887117463a8d7b46bb122b7b48" candidate_uuid: "3993e4208ef74160b16b6e96084a0698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e99679d3ba404a93ee74c1f52f0d67" is_pre_election: true
I20260926 23:28:26.406066 19998 raft_consensus.cc:2454] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3993e4208ef74160b16b6e96084a0698 in term 0.
I20260926 23:28:26.406226 19864 raft_consensus.cc:2454] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3993e4208ef74160b16b6e96084a0698 in term 0.
I20260926 23:28:26.406867 20065 leader_election.cc:302] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3993e4208ef74160b16b6e96084a0698, 51b26ca20d754ce5822512efaf778e6f; no voters:
I20260926 23:28:26.407325 20203 raft_consensus.cc:2781] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:28:26.407466 20203 raft_consensus.cc:482] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:28:26.407579 20203 raft_consensus.cc:3037] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:26.409806 20203 raft_consensus.cc:504] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.410387 20203 leader_election.cc:288] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [CANDIDATE]: Term 1 election: Requested vote from peers 32e99679d3ba404a93ee74c1f52f0d67 (127.0.97.1:46223), 51b26ca20d754ce5822512efaf778e6f (127.0.97.2:41745)
I20260926 23:28:26.410830 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2042e887117463a8d7b46bb122b7b48" candidate_uuid: "3993e4208ef74160b16b6e96084a0698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e99679d3ba404a93ee74c1f52f0d67"
I20260926 23:28:26.410897 19998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2042e887117463a8d7b46bb122b7b48" candidate_uuid: "3993e4208ef74160b16b6e96084a0698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b26ca20d754ce5822512efaf778e6f"
I20260926 23:28:26.411032 19864 raft_consensus.cc:3037] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:26.411166 19998 raft_consensus.cc:3037] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:26.413364 19864 raft_consensus.cc:2454] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3993e4208ef74160b16b6e96084a0698 in term 1.
I20260926 23:28:26.413693 19998 raft_consensus.cc:2454] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3993e4208ef74160b16b6e96084a0698 in term 1.
I20260926 23:28:26.413833 20068 leader_election.cc:302] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e99679d3ba404a93ee74c1f52f0d67, 3993e4208ef74160b16b6e96084a0698; no voters:
I20260926 23:28:26.414203 20203 raft_consensus.cc:2781] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:26.414685 20203 raft_consensus.cc:686] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 1 LEADER]: Becoming Leader. State: Replica: 3993e4208ef74160b16b6e96084a0698, State: Running, Role: LEADER
I20260926 23:28:26.415169 20203 consensus_queue.cc:237] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } }
I20260926 23:28:26.420094 19720 catalog_manager.cc:5495] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3993e4208ef74160b16b6e96084a0698 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3993e4208ef74160b16b6e96084a0698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3993e4208ef74160b16b6e96084a0698" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 } health_report { overall_health: UNKNOWN } } }
I20260926 23:28:26.449378 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.443769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.0.97.3:33551)
0926 23:28:26.444099 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.444109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.444185 (+ 76us) server_negotiation.cc:407] Connection header received
0926 23:28:26.444474 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.444479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.444564 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.444740 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:28:26.445133 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.445843 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.447168 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.447557 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.447650 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.448147 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:28:26.448739 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:28:26.448872 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.448976 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.449071 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
W20260926 23:28:26.453943 20178 tablet.cc:2369] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:26.458869 19864 raft_consensus.cc:1261] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 1 FOLLOWER]: Refusing update from remote peer 3993e4208ef74160b16b6e96084a0698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:26.458869 19998 raft_consensus.cc:1261] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 1 FOLLOWER]: Refusing update from remote peer 3993e4208ef74160b16b6e96084a0698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:26.460176 20206 consensus_queue.cc:1035] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e99679d3ba404a93ee74c1f52f0d67" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:26.460628 20203 consensus_queue.cc:1035] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b26ca20d754ce5822512efaf778e6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:26.472666 20209 mvcc.cc:205] Tried to move back new op lower bound from 7333745895243128832 to 7333745895080288256. Current Snapshot: MvccSnapshot[applied={T|T < 7333745895243128832}]
I20260926 23:28:26.489446 19720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:26.489792 19720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:26.494325 19720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b80674aa8fd4db08bc0c8923ac187f3: Sending DeleteTablet for 3 replicas of tablet c2042e887117463a8d7b46bb122b7b48
I20260926 23:28:26.495915 19844 tablet_service.cc:1514] Processing DeleteTablet for tablet c2042e887117463a8d7b46bb122b7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260926 23:28:26.496373 20112 tablet_service.cc:1514] Processing DeleteTablet for tablet c2042e887117463a8d7b46bb122b7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47762
I20260926 23:28:26.496395 19978 tablet_service.cc:1514] Processing DeleteTablet for tablet c2042e887117463a8d7b46bb122b7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40546
I20260926 23:28:26.498302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 19781
I20260926 23:28:26.511473 20219 tablet_replica.cc:330] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: stopping tablet replica
I20260926 23:28:26.512216 20219 raft_consensus.cc:2229] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:26.513023 20219 raft_consensus.cc:2258] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:26.519161 20220 tablet_replica.cc:330] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: stopping tablet replica
I20260926 23:28:26.519855 20220 raft_consensus.cc:2229] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:28:26.520782 20220 raft_consensus.cc:2258] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:26.523962 20219 ts_tablet_manager.cc:1905] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:28:26 dist-test-slave-q5g8 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:28:26.529863 20221 tablet_replica.cc:330] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: stopping tablet replica
I20260926 23:28:26.530347 20221 raft_consensus.cc:2229] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:26.530905 20221 raft_consensus.cc:2258] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:26.532888 20220 ts_tablet_manager.cc:1905] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:26.534322 20219 ts_tablet_manager.cc:1918] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:26.534514 20219 log.cc:1196] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/c2042e887117463a8d7b46bb122b7b48
I20260926 23:28:26.535079 20219 ts_tablet_manager.cc:1939] T c2042e887117463a8d7b46bb122b7b48 P 32e99679d3ba404a93ee74c1f52f0d67: Deleting consensus metadata
I20260926 23:28:26.537308 19707 catalog_manager.cc:4841] TS 32e99679d3ba404a93ee74c1f52f0d67 (127.0.97.1:46223): tablet c2042e887117463a8d7b46bb122b7b48 (table test-table [id=fa39686979484874b91859864ce62ff6]) successfully deleted
I20260926 23:28:26.539785 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.500355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.0.97.1:46223)
0926 23:28:26.500580 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.500587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.517588 (+ 17001us) server_negotiation.cc:407] Connection header received
0926 23:28:26.517891 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.517898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.518026 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.518143 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:26.520345 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.521578 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.523968 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.524744 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.524877 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.531146 (+ 6269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:26.531167 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:26.531187 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:26.531234 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:26.535552 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.536879 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.536891 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.536895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.537095 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.537754 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.537762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.537766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.538271 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:26.538578 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.539057 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.539347 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 23:28:26.540761 20221 ts_tablet_manager.cc:1905] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:26.541508 20220 ts_tablet_manager.cc:1918] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:26.541720 20220 log.cc:1196] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/c2042e887117463a8d7b46bb122b7b48
I20260926 23:28:26.542275 20220 ts_tablet_manager.cc:1939] T c2042e887117463a8d7b46bb122b7b48 P 3993e4208ef74160b16b6e96084a0698: Deleting consensus metadata
I20260926 23:28:26.542305 19904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:26.544224 19705 catalog_manager.cc:4841] TS 3993e4208ef74160b16b6e96084a0698 (127.0.97.3:33551): tablet c2042e887117463a8d7b46bb122b7b48 (table test-table [id=fa39686979484874b91859864ce62ff6]) successfully deleted
I20260926 23:28:26.548985 20221 ts_tablet_manager.cc:1918] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:26.549196 20221 log.cc:1196] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/c2042e887117463a8d7b46bb122b7b48
I20260926 23:28:26.549774 20221 ts_tablet_manager.cc:1939] T c2042e887117463a8d7b46bb122b7b48 P 51b26ca20d754ce5822512efaf778e6f: Deleting consensus metadata
I20260926 23:28:26.551993 19704 catalog_manager.cc:4841] TS 51b26ca20d754ce5822512efaf778e6f (127.0.97.2:41745): tablet c2042e887117463a8d7b46bb122b7b48 (table test-table [id=fa39686979484874b91859864ce62ff6]) successfully deleted
I20260926 23:28:26.641634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 19781
I20260926 23:28:26.669384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 19915
Sep 26 23:28:26 dist-test-slave-q5g8 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:28:26.687747 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.670440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.97.2:41745)
0926 23:28:26.670655 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.670662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.670687 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:28:26.670765 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.670770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.670867 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.671040 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:26.672398 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.673317 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.674757 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.675191 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.675343 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.680648 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:26.680668 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:26.680735 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:26.680793 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:26.684206 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.685253 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.685262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.685264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.685423 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.685887 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.685892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.685909 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.686359 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:26.686719 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.687073 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.687332 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:28:26.688915 20038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:26.787376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 19915
I20260926 23:28:26.812409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20049
Sep 26 23:28:26 dist-test-slave-q5g8 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:28:26.832062 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:26.813461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.97.3:33551)
0926 23:28:26.813667 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:26.813675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:26.813695 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:28:26.813768 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:26.813772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:26.813836 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:26.813916 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:26.815826 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.816908 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.818898 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:26.819398 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:26.819511 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:26.825392 (+ 5881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:26.825407 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:26.825425 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:26.825476 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:26.828754 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.829532 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.829545 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.829550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.829729 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:26.830317 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:26.830328 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:26.830331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:26.830742 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:26.831110 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:26.831399 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 23:28:26.831659 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:28:26.832937 20172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:26.963006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 20049
I20260926 23:28:26.983719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 19688
I20260926 23:28:26.984944 19750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:27.098888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 19688
2026-09-26T23:28:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4666 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T23:28:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:27Z Disabled control of system clock
I20260926 23:28:27.179019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37729
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:28:27.544085 20230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:27.544427 20230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:27.544524 20230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:27.552652 20230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:27.552824 20230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:27.552863 20230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:27.552915 20230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:27.552956 20230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:27.563107 20230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:27.564432 20230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:27.566741 20230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:27.579073 20238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:27.579099 20236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:27.579299 20235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:27.579780 20230 server_base.cc:1034] running on GCE node
I20260926 23:28:27.580660 20230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:27.582324 20230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:27.583598 20230 hybrid_clock.cc:648] HybridClock initialized: now 1790465307583520 us; error 59 us; skew 500 ppm
I20260926 23:28:27.584061 20230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:27.593443 20230 webserver.cc:460] Webserver started at http://127.0.97.62:40143/ using document root <none> and password file <none>
I20260926 23:28:27.594357 20230 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:27.594475 20230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:27.594879 20230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:27.598413 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "1fbee70125ed4fe4ab61978ea495afad"
format_stamp: "Formatted at 2026-09-26 23:28:27 on dist-test-slave-q5g8"
server_key: "d1f09fc7138628e01fb3cca88fc52519"
server_key_iv: "74506d9a1e8f1e1ce8cd62a4b98f575e"
server_key_version: "encryptionkey@0"
I20260926 23:28:27.599350 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "1fbee70125ed4fe4ab61978ea495afad"
format_stamp: "Formatted at 2026-09-26 23:28:27 on dist-test-slave-q5g8"
server_key: "d1f09fc7138628e01fb3cca88fc52519"
server_key_iv: "74506d9a1e8f1e1ce8cd62a4b98f575e"
server_key_version: "encryptionkey@0"
I20260926 23:28:27.605412 20230 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 23:28:27.609695 20244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:27.611282 20230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:28:27.611534 20230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "1fbee70125ed4fe4ab61978ea495afad"
format_stamp: "Formatted at 2026-09-26 23:28:27 on dist-test-slave-q5g8"
server_key: "d1f09fc7138628e01fb3cca88fc52519"
server_key_iv: "74506d9a1e8f1e1ce8cd62a4b98f575e"
server_key_version: "encryptionkey@0"
I20260926 23:28:27.611800 20230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:27.643961 20230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:27.644977 20230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:27.645318 20230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:27.667021 20295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37729 every 8 connection(s)
I20260926 23:28:27.667016 20230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37729
I20260926 23:28:27.668808 20230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:27.673051 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:27.677474 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 20230
I20260926 23:28:27.677656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:27.678120 388 external_mini_cluster.cc:1428] Setting key fbdab5ed39ac02ca3599e682a5ef0f33
I20260926 23:28:27.685657 20296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad: Bootstrap starting.
I20260926 23:28:27.689333 20296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:27.690662 20296 log.cc:824] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:27.694458 20296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad: No bootstrap required, opened a new log
I20260926 23:28:27.694566 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:27.680296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.0.97.62:37729)
0926 23:28:27.681516 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:27.681533 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:27.681605 (+ 72us) server_negotiation.cc:407] Connection header received
0926 23:28:27.684108 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:27.684138 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:27.684762 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:27.685246 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:27.685931 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:27.687338 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:27.688972 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:27.689975 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:27.690247 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:27.691670 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:27.691679 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:27.691699 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:27.692414 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:27.692509 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:27.692535 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:27.692723 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:27.692996 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 23:28:27.693394 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":409,"threads_started":1}
I20260926 23:28:27.704869 20296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbee70125ed4fe4ab61978ea495afad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37729 } }
I20260926 23:28:27.705416 20296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:27.705549 20296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbee70125ed4fe4ab61978ea495afad, State: Initialized, Role: FOLLOWER
I20260926 23:28:27.706332 20296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbee70125ed4fe4ab61978ea495afad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37729 } }
I20260926 23:28:27.706667 20296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:27.706843 20296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:27.707060 20296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:27.710587 20296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbee70125ed4fe4ab61978ea495afad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37729 } }
I20260926 23:28:27.711349 20296 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbee70125ed4fe4ab61978ea495afad; no voters:
I20260926 23:28:27.712138 20296 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:27.712345 20301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:27.713330 20301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [term 1 LEADER]: Becoming Leader. State: Replica: 1fbee70125ed4fe4ab61978ea495afad, State: Running, Role: LEADER
I20260926 23:28:27.714159 20301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbee70125ed4fe4ab61978ea495afad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37729 } }
I20260926 23:28:27.714819 20296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:27.717985 20303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbee70125ed4fe4ab61978ea495afad. Latest consensus state: current_term: 1 leader_uuid: "1fbee70125ed4fe4ab61978ea495afad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbee70125ed4fe4ab61978ea495afad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37729 } } }
I20260926 23:28:27.718519 20303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:27.719225 20302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fbee70125ed4fe4ab61978ea495afad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbee70125ed4fe4ab61978ea495afad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37729 } } }
I20260926 23:28:27.719610 20302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:27.719944 20308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:27.728713 20308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:27.747124 20308 catalog_manager.cc:1347] Generated new cluster ID: 23e49f0aac4d4c4aa33e3051e75d27a6
I20260926 23:28:27.747279 20308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:27.775539 20308 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:27.777290 20308 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:27.799036 20308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad: Generated new TSK 0
I20260926 23:28:27.800139 20308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:27.821034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37729
--builtin_ntp_servers=127.0.97.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:28:28.243813 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:28.244217 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:28.244365 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:28.252223 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:28.252357 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:28.252477 20321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:28:28.263315 20321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:28.264823 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:28.266774 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:28.282985 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:28.283121 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:28.283115 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:28.283921 20321 server_base.cc:1034] running on GCE node
I20260926 23:28:28.284859 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:28.286120 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:28.287431 20321 hybrid_clock.cc:648] HybridClock initialized: now 1790465308287362 us; error 79 us; skew 500 ppm
I20260926 23:28:28.287843 20321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:28.290591 20321 webserver.cc:460] Webserver started at http://127.0.97.1:37183/ using document root <none> and password file <none>
I20260926 23:28:28.291395 20321 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:28.291553 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:28.291921 20321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:28.295095 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data/instance:
uuid: "fbaafb7f0a1946f89de9a7498666cf18"
format_stamp: "Formatted at 2026-09-26 23:28:28 on dist-test-slave-q5g8"
server_key: "796eaa465ab71992b50d3dab50e5df0f"
server_key_iv: "8edb8b67e9c49170df8179d6769c1eb3"
server_key_version: "encryptionkey@0"
I20260926 23:28:28.295977 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fbaafb7f0a1946f89de9a7498666cf18"
format_stamp: "Formatted at 2026-09-26 23:28:28 on dist-test-slave-q5g8"
server_key: "796eaa465ab71992b50d3dab50e5df0f"
server_key_iv: "8edb8b67e9c49170df8179d6769c1eb3"
server_key_version: "encryptionkey@0"
I20260926 23:28:28.301676 20321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 23:28:28.305557 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:28.307083 20321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:28:28.307294 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal
uuid: "fbaafb7f0a1946f89de9a7498666cf18"
format_stamp: "Formatted at 2026-09-26 23:28:28 on dist-test-slave-q5g8"
server_key: "796eaa465ab71992b50d3dab50e5df0f"
server_key_iv: "8edb8b67e9c49170df8179d6769c1eb3"
server_key_version: "encryptionkey@0"
I20260926 23:28:28.307551 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:28.328811 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:28.329849 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:28.330199 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:28.331364 20321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:28.333181 20321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:28.333324 20321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:28.333479 20321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:28.333578 20321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:28.369269 20321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41883
I20260926 23:28:28.369329 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41883 every 8 connection(s)
I20260926 23:28:28.370750 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:28:28.377102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 20321
I20260926 23:28:28.377298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:28:28.377775 388 external_mini_cluster.cc:1428] Setting key 5344806c709d33b89f2717817acff525
I20260926 23:28:28.381294 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37729
--builtin_ntp_servers=127.0.97.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:28:28.387004 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:28.373786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42109 (local address 127.0.97.62:37729)
0926 23:28:28.374294 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:28.374299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:28.375840 (+ 1541us) server_negotiation.cc:407] Connection header received
0926 23:28:28.378164 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:28.378171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:28.378277 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:28.378437 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:28.379834 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.381106 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.383191 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.383964 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.384143 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:28.385875 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:28.385884 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:28.385892 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:28.385924 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:28.385999 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:28.386019 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:28.386127 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:28.386377 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 23:28:28.386512 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260926 23:28:28.388020 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:28.374496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37729 (local address 127.0.97.1:42109)
0926 23:28:28.375645 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:28.375687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:28.377741 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:28.378782 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:28.378792 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:28.378916 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:28.379606 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:28.379658 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.381344 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.381349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:28.382976 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:28.382992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.384247 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.384252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:28.384505 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:28.385477 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:28.385480 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:28.385528 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:28.385628 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:28.386138 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:28.386148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:28.386155 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:28:28.386323 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 23:28:28.386721 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":426,"threads_started":1}
I20260926 23:28:28.390081 20448 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37729
I20260926 23:28:28.390519 20448 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:28.391580 20448 heartbeater.cc:511] Master 127.0.97.62:37729 requested a full tablet report, sending...
I20260926 23:28:28.394408 20261 ts_manager.cc:194] Registered new tserver with Master: fbaafb7f0a1946f89de9a7498666cf18 (127.0.97.1:41883)
I20260926 23:28:28.396899 20261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42109
W20260926 23:28:28.750061 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:28.750401 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:28.750492 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:28.758517 20452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:28.758644 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:28.758756 20452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:28:28.768903 20452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:28.770364 20452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:28.772373 20452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:28.787258 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:28.787456 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:28.787252 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:28.788741 20452 server_base.cc:1034] running on GCE node
I20260926 23:28:28.789506 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:28.790745 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:28.792042 20452 hybrid_clock.cc:648] HybridClock initialized: now 1790465308791985 us; error 57 us; skew 500 ppm
I20260926 23:28:28.792462 20452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:28.795622 20452 webserver.cc:460] Webserver started at http://127.0.97.2:41339/ using document root <none> and password file <none>
I20260926 23:28:28.796527 20452 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:28.796737 20452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:28.797166 20452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:28.800595 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data/instance:
uuid: "d711cf80e0f94505a1c2dfd0dc72ad16"
format_stamp: "Formatted at 2026-09-26 23:28:28 on dist-test-slave-q5g8"
server_key: "4152a174ebd45f1c8ca53d165921e46a"
server_key_iv: "f2d35066694b9ac46895f75ccf608a57"
server_key_version: "encryptionkey@0"
I20260926 23:28:28.801688 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d711cf80e0f94505a1c2dfd0dc72ad16"
format_stamp: "Formatted at 2026-09-26 23:28:28 on dist-test-slave-q5g8"
server_key: "4152a174ebd45f1c8ca53d165921e46a"
server_key_iv: "f2d35066694b9ac46895f75ccf608a57"
server_key_version: "encryptionkey@0"
I20260926 23:28:28.807601 20452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:28:28.811745 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:28.813688 20452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:28:28.813933 20452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal
uuid: "d711cf80e0f94505a1c2dfd0dc72ad16"
format_stamp: "Formatted at 2026-09-26 23:28:28 on dist-test-slave-q5g8"
server_key: "4152a174ebd45f1c8ca53d165921e46a"
server_key_iv: "f2d35066694b9ac46895f75ccf608a57"
server_key_version: "encryptionkey@0"
I20260926 23:28:28.814205 20452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:28.841848 20452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:28.842866 20452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:28.843227 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:28.844560 20452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:28.846342 20452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:28.846509 20452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:28.846673 20452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:28.846779 20452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:28.884603 20452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37379
I20260926 23:28:28.884791 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37379 every 8 connection(s)
I20260926 23:28:28.886181 20452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:28:28.895313 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 20452
I20260926 23:28:28.895499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:28:28.895967 388 external_mini_cluster.cc:1428] Setting key 6b788b5ec1fe7536a68f173c730bce40
I20260926 23:28:28.899269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37729
--builtin_ntp_servers=127.0.97.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:28:28.901600 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:28.889235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59195 (local address 127.0.97.62:37729)
0926 23:28:28.889721 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:28.889726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:28.890915 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 23:28:28.893019 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:28.893024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:28.893123 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:28.893250 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:28.894429 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.895454 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.897554 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.898199 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.898367 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:28.900260 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:28.900271 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:28.900280 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:28.900311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:28.900373 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:28.900396 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:28.900511 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:28.900937 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 23:28:28.901080 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":260,"threads_started":1}
I20260926 23:28:28.902797 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:28.889798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37729 (local address 127.0.97.2:59195)
0926 23:28:28.890736 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:28.890773 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:28.892602 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:28.893509 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:28.893519 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:28.893627 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:28.894235 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:28.894252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.895663 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.895668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:28.897305 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:28.897322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:28.898532 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:28.898539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:28.898793 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:28.899767 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:28.899770 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:28.899882 (+ 112us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:28.899995 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:28.900552 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:28.900569 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:28.900581 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 23:28:28.900902 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 23:28:28.901338 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":281,"threads_started":1}
I20260926 23:28:28.904760 20579 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37729
I20260926 23:28:28.905206 20579 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:28.906338 20579 heartbeater.cc:511] Master 127.0.97.62:37729 requested a full tablet report, sending...
I20260926 23:28:28.908404 20261 ts_manager.cc:194] Registered new tserver with Master: d711cf80e0f94505a1c2dfd0dc72ad16 (127.0.97.2:37379)
I20260926 23:28:28.909512 20261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59195
W20260926 23:28:29.279273 20583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:29.279619 20583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:29.279889 20583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:29.288141 20583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:29.288285 20583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:29.288452 20583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:28:29.298970 20583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:29.300518 20583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:29.302678 20583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:29.319128 20588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:29.319921 20589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:29.321385 20591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:29.321933 20583 server_base.cc:1034] running on GCE node
I20260926 23:28:29.323139 20583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:29.324962 20583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:29.326408 20583 hybrid_clock.cc:648] HybridClock initialized: now 1790465309326328 us; error 110 us; skew 500 ppm
I20260926 23:28:29.327049 20583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:28:29.331410 20583 webserver.cc:460] Webserver started at http://127.0.97.3:34323/ using document root <none> and password file <none>
I20260926 23:28:29.332875 20583 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:29.333066 20583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:29.333601 20583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:29.339370 20583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data/instance:
uuid: "70766ef204ea4962b66770c2b46a9d11"
format_stamp: "Formatted at 2026-09-26 23:28:29 on dist-test-slave-q5g8"
server_key: "e939c605d1961151b440ddb8573427f8"
server_key_iv: "09494d49f6071cb5bd607336200906ff"
server_key_version: "encryptionkey@0"
I20260926 23:28:29.340943 20583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal/instance:
uuid: "70766ef204ea4962b66770c2b46a9d11"
format_stamp: "Formatted at 2026-09-26 23:28:29 on dist-test-slave-q5g8"
server_key: "e939c605d1961151b440ddb8573427f8"
server_key_iv: "09494d49f6071cb5bd607336200906ff"
server_key_version: "encryptionkey@0"
I20260926 23:28:29.351049 20583 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260926 23:28:29.356009 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:29.357847 20583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 23:28:29.358165 20583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal
uuid: "70766ef204ea4962b66770c2b46a9d11"
format_stamp: "Formatted at 2026-09-26 23:28:29 on dist-test-slave-q5g8"
server_key: "e939c605d1961151b440ddb8573427f8"
server_key_iv: "09494d49f6071cb5bd607336200906ff"
server_key_version: "encryptionkey@0"
I20260926 23:28:29.358461 20583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:29.382493 20583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:29.383584 20583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:29.383932 20583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:29.385205 20583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:28:29.387281 20583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:28:29.387423 20583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:29.387580 20583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:28:29.387676 20583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:29.401911 20448 heartbeater.cc:503] Master 127.0.97.62:37729 was elected leader, sending a full tablet report...
I20260926 23:28:29.427616 20583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43461
I20260926 23:28:29.427736 20709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43461 every 8 connection(s)
I20260926 23:28:29.429286 20583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:28:29.432417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 20583
I20260926 23:28:29.432607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:28:29.433403 388 external_mini_cluster.cc:1428] Setting key c313ec2ffbbc3b7b9e6af7927d1e0dd2
I20260926 23:28:29.443967 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.432408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53175 (local address 127.0.97.62:37729)
0926 23:28:29.434897 (+ 2489us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.434903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.434926 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:28:29.436956 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.436962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.437056 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.437176 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:29.438520 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.439280 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.441059 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.441485 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.441623 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.443172 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:29.443177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:29.443180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:29.443201 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:29.443243 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:29.443255 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:29.443328 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.443570 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.443676 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2374,"thread_start_us":2226,"threads_started":1}
I20260926 23:28:29.445286 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.433091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37729 (local address 127.0.97.3:53175)
0926 23:28:29.434409 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:29.434468 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:29.436537 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:29.437543 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:29.437553 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:29.437661 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:28:29.438330 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.438350 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.439443 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.439447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.440895 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.440911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.441729 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.441734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.441971 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.442795 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:28:29.442798 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:28:29.442862 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:28:29.442964 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:28:29.443350 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:28:29.443359 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:28:29.443365 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:28:29.443540 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 23:28:29.443908 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":488,"threads_started":1}
I20260926 23:28:29.446883 20710 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37729
I20260926 23:28:29.447232 20710 heartbeater.cc:464] Registering TS with master...
I20260926 23:28:29.448220 20710 heartbeater.cc:511] Master 127.0.97.62:37729 requested a full tablet report, sending...
I20260926 23:28:29.450001 20261 ts_manager.cc:194] Registered new tserver with Master: 70766ef204ea4962b66770c2b46a9d11 (127.0.97.3:43461)
I20260926 23:28:29.450802 20261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:53175
I20260926 23:28:29.462630 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:28:29.473225 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.467337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.0.97.62:37729)
0926 23:28:29.467527 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.467531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.467864 (+ 333us) server_negotiation.cc:407] Connection header received
0926 23:28:29.468015 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.468018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.468085 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.468181 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:29.468650 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.469492 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.470936 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.471351 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.471501 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.472529 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:29.472534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:29.472537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:29.472558 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:29.472590 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:29.472602 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:29.472675 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.472893 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.472980 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:28:29.477581 20261 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:28:29.485428 20261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:28:29.513805 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.499560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37379 (local address 127.0.0.1:53346)
0926 23:28:29.501447 (+ 1887us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:29.501475 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:29.501650 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:29.504657 (+ 3007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:29.504662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:29.504722 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:29.505429 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.505443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.507500 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.507507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.510376 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.510391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.512706 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.512714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.512897 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.512943 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 23:28:29.513049 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:28:29.513299 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"mutex_wait_us":128,"thread_start_us":836,"threads_started":1}
I20260926 23:28:29.513878 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.500288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.97.1:41883)
0926 23:28:29.504232 (+ 3944us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.504243 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.504273 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:28:29.504371 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.504376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.504596 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.505017 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:29.505827 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.507292 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.510598 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.512311 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.512490 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.512907 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.513287 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.513493 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3764,"thread_start_us":3614,"threads_started":1}
I20260926 23:28:29.513881 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.499531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41883 (local address 127.0.0.1:38248)
0926 23:28:29.500479 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:29.500536 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:29.500754 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:29.504886 (+ 4132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:29.504894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:29.504923 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:29.505576 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.505591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.507453 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.507461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.510270 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.510293 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.512546 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.512553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.512830 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.512919 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 23:28:29.513240 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 23:28:29.513397 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":489,"thread_start_us":1535,"threads_started":1}
I20260926 23:28:29.515620 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.500987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.0.97.3:43461)
0926 23:28:29.502367 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.502381 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.503098 (+ 717us) server_negotiation.cc:407] Connection header received
0926 23:28:29.503450 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.503460 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.503775 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.503988 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:29.505803 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.507373 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.510646 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.512504 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.512646 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.512987 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.515008 (+ 2021us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.515190 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":999,"threads_started":1}
I20260926 23:28:29.516115 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.500157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43461 (local address 127.0.0.1:60220)
0926 23:28:29.502942 (+ 2785us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:29.502981 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:29.503226 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:29.504303 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:29.504313 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:29.504347 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:29.504870 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.504888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.507519 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.507530 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.510405 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.510420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.513119 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.513125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.513336 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.513366 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:28:29.513444 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:28:29.513573 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2624,"mutex_wait_us":302,"thread_start_us":2443,"threads_started":1}
I20260926 23:28:29.516969 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.500198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.0.97.2:37379)
0926 23:28:29.503967 (+ 3769us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.503976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.504009 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:28:29.504112 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.504122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.504405 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.504645 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:29.505625 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.507309 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.510664 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.512402 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.514463 (+ 2061us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.514755 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.514876 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.515088 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3537,"thread_start_us":3372,"threads_started":1}
I20260926 23:28:29.519222 20383 tablet_service.cc:1467] Processing CreateTablet for tablet 8692574cd74c48ae820487e84197e969 (DEFAULT_TABLE table=test-table [id=a46c24e68c9c43f0828041e6f70165c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:29.521664 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8692574cd74c48ae820487e84197e969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:29.521127 20645 tablet_service.cc:1467] Processing CreateTablet for tablet 8692574cd74c48ae820487e84197e969 (DEFAULT_TABLE table=test-table [id=a46c24e68c9c43f0828041e6f70165c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:29.523001 20645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8692574cd74c48ae820487e84197e969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:29.522183 20514 tablet_service.cc:1467] Processing CreateTablet for tablet 8692574cd74c48ae820487e84197e969 (DEFAULT_TABLE table=test-table [id=a46c24e68c9c43f0828041e6f70165c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:28:29.524219 20514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8692574cd74c48ae820487e84197e969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:29.539963 20729 tablet_bootstrap.cc:492] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Bootstrap starting.
I20260926 23:28:29.542869 20730 tablet_bootstrap.cc:492] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Bootstrap starting.
I20260926 23:28:29.544350 20729 tablet_bootstrap.cc:654] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:29.546078 20729 log.cc:824] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:29.547067 20730 tablet_bootstrap.cc:654] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:29.548108 20731 tablet_bootstrap.cc:492] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Bootstrap starting.
I20260926 23:28:29.548549 20730 log.cc:824] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:29.554222 20731 tablet_bootstrap.cc:654] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:29.555516 20729 tablet_bootstrap.cc:492] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: No bootstrap required, opened a new log
I20260926 23:28:29.555886 20729 ts_tablet_manager.cc:1397] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260926 23:28:29.555876 20731 log.cc:824] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:29.559197 20730 tablet_bootstrap.cc:492] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: No bootstrap required, opened a new log
I20260926 23:28:29.559629 20730 ts_tablet_manager.cc:1397] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260926 23:28:29.571249 20729 raft_consensus.cc:348] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.571900 20729 raft_consensus.cc:374] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:29.572021 20729 raft_consensus.cc:729] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaafb7f0a1946f89de9a7498666cf18, State: Initialized, Role: FOLLOWER
I20260926 23:28:29.572904 20731 tablet_bootstrap.cc:492] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: No bootstrap required, opened a new log
I20260926 23:28:29.572988 20729 consensus_queue.cc:260] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.575714 20731 ts_tablet_manager.cc:1397] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.012s
I20260926 23:28:29.575568 20730 raft_consensus.cc:348] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.576166 20730 raft_consensus.cc:374] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:29.576300 20730 raft_consensus.cc:729] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d711cf80e0f94505a1c2dfd0dc72ad16, State: Initialized, Role: FOLLOWER
I20260926 23:28:29.577247 20730 consensus_queue.cc:260] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.580051 20729 ts_tablet_manager.cc:1428] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Time spent starting tablet: real 0.024s user 0.021s sys 0.000s
I20260926 23:28:29.585027 20579 heartbeater.cc:503] Master 127.0.97.62:37729 was elected leader, sending a full tablet report...
I20260926 23:28:29.586282 20730 ts_tablet_manager.cc:1428] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Time spent starting tablet: real 0.026s user 0.022s sys 0.004s
I20260926 23:28:29.590579 20731 raft_consensus.cc:348] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.591429 20731 raft_consensus.cc:374] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:29.591634 20731 raft_consensus.cc:729] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70766ef204ea4962b66770c2b46a9d11, State: Initialized, Role: FOLLOWER
I20260926 23:28:29.592651 20731 consensus_queue.cc:260] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.594373 20710 heartbeater.cc:503] Master 127.0.97.62:37729 was elected leader, sending a full tablet report...
I20260926 23:28:29.595456 20731 ts_tablet_manager.cc:1428] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Time spent starting tablet: real 0.019s user 0.014s sys 0.002s
W20260926 23:28:29.623551 20449 tablet.cc:2369] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:28:29.638545 20580 tablet.cc:2369] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:28:29.681267 20711 tablet.cc:2369] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:28:29.881181 20736 raft_consensus.cc:482] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:28:29.881521 20736 raft_consensus.cc:504] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.883406 20736 leader_election.cc:288] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbaafb7f0a1946f89de9a7498666cf18 (127.0.97.1:41883), 70766ef204ea4962b66770c2b46a9d11 (127.0.97.3:43461)
I20260926 23:28:29.892900 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.883988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47733 (local address 127.0.97.1:41883)
0926 23:28:29.884248 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.884260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.885752 (+ 1492us) server_negotiation.cc:407] Connection header received
0926 23:28:29.886053 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.886062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.886160 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.886263 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:29.887124 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.887949 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.890644 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.891615 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.891803 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.891938 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.892457 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.892601 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 23:28:29.893095 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.883785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41883 (local address 127.0.97.2:47733)
0926 23:28:29.885616 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:29.885632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:29.885815 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:29.886313 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:29.886327 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:29.886373 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:29.886917 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.886927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.888121 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.888134 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.890395 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.890412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.891874 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.891882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.892153 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.892286 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 23:28:29.892410 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 23:28:29.892576 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"mutex_wait_us":105,"thread_start_us":900,"threads_started":1}
I20260926 23:28:29.893258 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.884465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56195 (local address 127.0.97.3:43461)
0926 23:28:29.884737 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.884744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.885157 (+ 413us) server_negotiation.cc:407] Connection header received
0926 23:28:29.885588 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.885595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.885678 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.885778 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:28:29.886649 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.887742 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.890657 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.891866 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.891963 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.892058 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.892781 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.892938 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 23:28:29.894024 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.883676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43461 (local address 127.0.97.2:56195)
0926 23:28:29.885010 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0926 23:28:29.885034 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:28:29.885357 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:28:29.885867 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:28:29.885876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:28:29.885908 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:28:29.886436 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.886459 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.887925 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.887931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.890434 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:28:29.890450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.892165 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.892173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:28:29.892491 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.892529 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:28:29.893470 (+ 941us) client_negotiation.cc:240] Negotiation successful
0926 23:28:29.893620 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":111,"thread_start_us":1565,"threads_started":1}
I20260926 23:28:29.894505 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692574cd74c48ae820487e84197e969" candidate_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaafb7f0a1946f89de9a7498666cf18" is_pre_election: true
I20260926 23:28:29.895205 20403 raft_consensus.cc:2454] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d711cf80e0f94505a1c2dfd0dc72ad16 in term 0.
I20260926 23:28:29.895185 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692574cd74c48ae820487e84197e969" candidate_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70766ef204ea4962b66770c2b46a9d11" is_pre_election: true
I20260926 23:28:29.895676 20665 raft_consensus.cc:2454] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d711cf80e0f94505a1c2dfd0dc72ad16 in term 0.
I20260926 23:28:29.896117 20467 leader_election.cc:302] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d711cf80e0f94505a1c2dfd0dc72ad16, fbaafb7f0a1946f89de9a7498666cf18; no voters:
I20260926 23:28:29.896662 20736 raft_consensus.cc:2781] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:28:29.896919 20736 raft_consensus.cc:482] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:28:29.897089 20736 raft_consensus.cc:3037] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:29.899854 20736 raft_consensus.cc:504] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.900767 20736 leader_election.cc:288] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [CANDIDATE]: Term 1 election: Requested vote from peers fbaafb7f0a1946f89de9a7498666cf18 (127.0.97.1:41883), 70766ef204ea4962b66770c2b46a9d11 (127.0.97.3:43461)
I20260926 23:28:29.901240 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692574cd74c48ae820487e84197e969" candidate_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaafb7f0a1946f89de9a7498666cf18"
I20260926 23:28:29.901258 20665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692574cd74c48ae820487e84197e969" candidate_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70766ef204ea4962b66770c2b46a9d11"
I20260926 23:28:29.901510 20665 raft_consensus.cc:3037] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:29.901502 20403 raft_consensus.cc:3037] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:29.903873 20665 raft_consensus.cc:2454] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d711cf80e0f94505a1c2dfd0dc72ad16 in term 1.
I20260926 23:28:29.904330 20468 leader_election.cc:302] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70766ef204ea4962b66770c2b46a9d11, d711cf80e0f94505a1c2dfd0dc72ad16; no voters:
I20260926 23:28:29.904533 20403 raft_consensus.cc:2454] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d711cf80e0f94505a1c2dfd0dc72ad16 in term 1.
I20260926 23:28:29.904657 20736 raft_consensus.cc:2781] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:29.905306 20736 raft_consensus.cc:686] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 1 LEADER]: Becoming Leader. State: Replica: d711cf80e0f94505a1c2dfd0dc72ad16, State: Running, Role: LEADER
I20260926 23:28:29.905804 20736 consensus_queue.cc:237] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } }
I20260926 23:28:29.910723 20261 catalog_manager.cc:5495] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 reported cstate change: term changed from 0 to 1, leader changed from <none> to d711cf80e0f94505a1c2dfd0dc72ad16 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d711cf80e0f94505a1c2dfd0dc72ad16" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 } health_report { overall_health: UNKNOWN } } }
I20260926 23:28:29.979839 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:29.974005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.0.97.2:37379)
0926 23:28:29.974268 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:29.974277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:29.974668 (+ 391us) server_negotiation.cc:407] Connection header received
0926 23:28:29.974984 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:29.974988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:29.975077 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:29.975203 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:28:29.975617 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.976288 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.977803 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:29.978193 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:29.978287 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:29.978721 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:28:29.979228 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:28:29.979342 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:29.979459 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 23:28:29.979523 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:28:29.988238 20665 raft_consensus.cc:1261] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 1 FOLLOWER]: Refusing update from remote peer d711cf80e0f94505a1c2dfd0dc72ad16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:29.988511 20403 raft_consensus.cc:1261] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 1 FOLLOWER]: Refusing update from remote peer d711cf80e0f94505a1c2dfd0dc72ad16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:28:29.989370 20740 consensus_queue.cc:1035] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70766ef204ea4962b66770c2b46a9d11" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:29.989847 20736 consensus_queue.cc:1035] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbaafb7f0a1946f89de9a7498666cf18" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:28:30.014940 20261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:30.015208 20261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:28:30.018859 20261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fbee70125ed4fe4ab61978ea495afad: Sending DeleteTablet for 3 replicas of tablet 8692574cd74c48ae820487e84197e969
I20260926 23:28:30.020156 20514 tablet_service.cc:1514] Processing DeleteTablet for tablet 8692574cd74c48ae820487e84197e969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:30 UTC) from {username='slave'} at 127.0.0.1:53346
I20260926 23:28:30.020478 20645 tablet_service.cc:1514] Processing DeleteTablet for tablet 8692574cd74c48ae820487e84197e969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:30 UTC) from {username='slave'} at 127.0.0.1:60220
I20260926 23:28:30.020504 20383 tablet_service.cc:1514] Processing DeleteTablet for tablet 8692574cd74c48ae820487e84197e969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:28:30 UTC) from {username='slave'} at 127.0.0.1:38248
I20260926 23:28:30.022612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20321
I20260926 23:28:30.033245 20755 tablet_replica.cc:330] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: stopping tablet replica
I20260926 23:28:30.034020 20755 raft_consensus.cc:2229] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:30.034694 20755 raft_consensus.cc:2258] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:30.035652 20756 tablet_replica.cc:330] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: stopping tablet replica
I20260926 23:28:30.036219 20756 raft_consensus.cc:2229] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:28:30.036903 20756 raft_consensus.cc:2258] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:30.037251 20754 tablet_replica.cc:330] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: stopping tablet replica
I20260926 23:28:30.037693 20754 raft_consensus.cc:2229] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:28:30.038401 20754 raft_consensus.cc:2258] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:28:30.039654 20756 ts_tablet_manager.cc:1905] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:30.040635 20754 ts_tablet_manager.cc:1905] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:30.042074 20755 ts_tablet_manager.cc:1905] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:28:30.047107 20756 ts_tablet_manager.cc:1918] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:30.047303 20756 log.cc:1196] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-0/wal/wals/8692574cd74c48ae820487e84197e969
I20260926 23:28:30.047849 20756 ts_tablet_manager.cc:1939] T 8692574cd74c48ae820487e84197e969 P fbaafb7f0a1946f89de9a7498666cf18: Deleting consensus metadata
I20260926 23:28:30.048413 20754 ts_tablet_manager.cc:1918] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:30.048611 20754 log.cc:1196] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-1/wal/wals/8692574cd74c48ae820487e84197e969
I20260926 23:28:30.049340 20754 ts_tablet_manager.cc:1939] T 8692574cd74c48ae820487e84197e969 P d711cf80e0f94505a1c2dfd0dc72ad16: Deleting consensus metadata
I20260926 23:28:30.050285 20245 catalog_manager.cc:4841] TS fbaafb7f0a1946f89de9a7498666cf18 (127.0.97.1:41883): tablet 8692574cd74c48ae820487e84197e969 (table test-table [id=a46c24e68c9c43f0828041e6f70165c3]) successfully deleted
I20260926 23:28:30.051187 20248 catalog_manager.cc:4841] TS d711cf80e0f94505a1c2dfd0dc72ad16 (127.0.97.2:37379): tablet 8692574cd74c48ae820487e84197e969 (table test-table [id=a46c24e68c9c43f0828041e6f70165c3]) successfully deleted
I20260926 23:28:30.052148 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:30.024233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.97.1:41883)
0926 23:28:30.024476 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:30.024485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:30.037368 (+ 12883us) server_negotiation.cc:407] Connection header received
0926 23:28:30.037734 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:30.037741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:30.037869 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:30.037970 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:30.038750 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:30.039878 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:30.045632 (+ 5754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:30.046373 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:30.046490 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:30.049172 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:30.049185 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:30.049204 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:30.050054 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:30.050127 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:30.050155 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:30.050248 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:30.051606 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0926 23:28:30.051771 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:28:30.052914 20443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:30.058262 20755 ts_tablet_manager.cc:1918] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:28:30.058540 20755 log.cc:1196] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Deleting WAL directory at /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790464978839002-388-0/minicluster-data/ts-2/wal/wals/8692574cd74c48ae820487e84197e969
I20260926 23:28:30.059229 20755 ts_tablet_manager.cc:1939] T 8692574cd74c48ae820487e84197e969 P 70766ef204ea4962b66770c2b46a9d11: Deleting consensus metadata
I20260926 23:28:30.061866 20246 catalog_manager.cc:4841] TS 70766ef204ea4962b66770c2b46a9d11 (127.0.97.3:43461): tablet 8692574cd74c48ae820487e84197e969 (table test-table [id=a46c24e68c9c43f0828041e6f70165c3]) successfully deleted
I20260926 23:28:30.153160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 20321
I20260926 23:28:30.179625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20452
I20260926 23:28:30.187866 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:30.180556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.0.97.2:37379)
0926 23:28:30.180786 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:30.180791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:30.180811 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:28:30.181003 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:30.181007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:30.181099 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:30.181208 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:30.181706 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:30.182586 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:30.184034 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:30.184497 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:30.184680 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:30.186171 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:30.186181 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:30.186199 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:30.186916 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:30.186973 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:30.187008 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:30.187159 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:30.187334 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 23:28:30.187459 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 23:28:30.188673 20574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:30.308243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 20452
I20260926 23:28:30.332970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20583
I20260926 23:28:30.339939 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:30.333889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.97.3:43461)
0926 23:28:30.334069 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:30.334074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:30.334093 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:28:30.334156 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:30.334160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:30.334247 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:30.334348 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:30.334720 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:30.335492 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:30.336986 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:30.337386 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:30.337500 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:30.338552 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:30.338560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:30.338576 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:30.339210 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:30.339257 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:30.339278 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:30.339368 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:30.339496 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 23:28:30.339595 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:28:30.340673 20705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:30.432655 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 20583
I20260926 23:28:30.449667 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20230
I20260926 23:28:30.450917 20291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:30.565984 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 20230
2026-09-26T23:28:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3470 ms)
[----------] 6 tests from EncryptionPolicyTest (23536 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:30 dist-test-slave-q5g8 krb5kdc[20764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:30 dist-test-slave-q5g8 krb5kdc[20764](info): set up 2 sockets
Sep 26 23:28:30 dist-test-slave-q5g8 krb5kdc[20764](info): commencing operation
krb5kdc: starting...
W20260926 23:28:31.725468 388 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:31 dist-test-slave-q5g8 krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:32.005082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:32.379308 20777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:32.379664 20777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:32.379765 20777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:32.387753 20777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:32.387912 20777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:32.387962 20777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:32.388021 20777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:32.388068 20777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:32.399046 20777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:32.400540 20777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:32.402135 20777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:32.415913 20782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:32.416404 20783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:32.416708 20785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:32.417531 20777 server_base.cc:1034] running on GCE node
I20260926 23:28:32.418771 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:32.421178 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:32.422468 20777 hybrid_clock.cc:648] HybridClock initialized: now 1790465312422416 us; error 87 us; skew 500 ppm
I20260926 23:28:32.422983 20777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:32 dist-test-slave-q5g8 krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:32.426753 20777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:32.428566 20777 webserver.cc:460] Webserver started at http://127.0.0.1:38863/ using document root <none> and password file <none>
I20260926 23:28:32.429415 20777 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:32.429536 20777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:32.429837 20777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:32.433434 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "1c3c6d361f674f0bbd0700cd773d427a"
format_stamp: "Formatted at 2026-09-26 23:28:32 on dist-test-slave-q5g8"
server_key: "e15f504cf8414cfc84faa6fb54543ea8"
server_key_iv: "4cdbefbb79211bb17239b7e90a48d43a"
server_key_version: "encryptionkey@0"
I20260926 23:28:32.434299 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c3c6d361f674f0bbd0700cd773d427a"
format_stamp: "Formatted at 2026-09-26 23:28:32 on dist-test-slave-q5g8"
server_key: "e15f504cf8414cfc84faa6fb54543ea8"
server_key_iv: "4cdbefbb79211bb17239b7e90a48d43a"
server_key_version: "encryptionkey@0"
I20260926 23:28:32.440551 20777 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:28:32.445340 20792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:32.447093 20777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260926 23:28:32.447348 20777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "1c3c6d361f674f0bbd0700cd773d427a"
format_stamp: "Formatted at 2026-09-26 23:28:32 on dist-test-slave-q5g8"
server_key: "e15f504cf8414cfc84faa6fb54543ea8"
server_key_iv: "4cdbefbb79211bb17239b7e90a48d43a"
server_key_version: "encryptionkey@0"
I20260926 23:28:32.447600 20777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:32.484195 20777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:32.489456 20777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:32.489842 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:32.512279 20777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45947
I20260926 23:28:32.512413 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45947 every 8 connection(s)
I20260926 23:28:32.514127 20777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:32.518831 20844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:32.524546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 20777
I20260926 23:28:32.524850 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:32.525583 388 external_mini_cluster.cc:1428] Setting key cb757a66d26b66d6aed08cd17e7e1482
I20260926 23:28:32.531741 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a: Bootstrap starting.
I20260926 23:28:32.536312 20844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:32.537900 20844 log.cc:824] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:32.542225 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a: No bootstrap required, opened a new log
Sep 26 23:28:32 dist-test-slave-q5g8 krb5kdc[20764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:28:32.551822 20844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c6d361f674f0bbd0700cd773d427a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260926 23:28:32.552510 20844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:32.552747 20844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3c6d361f674f0bbd0700cd773d427a, State: Initialized, Role: FOLLOWER
I20260926 23:28:32.553637 20844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c6d361f674f0bbd0700cd773d427a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260926 23:28:32.553960 20844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:32.554163 20844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:32.554395 20844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:32.554797 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:32.527980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46925 (local address 127.0.0.1:45947)
0926 23:28:32.529050 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:32.529064 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:32.529122 (+ 58us) server_negotiation.cc:407] Connection header received
0926 23:28:32.531096 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:32.531126 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:32.531636 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:32.532145 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:32.534317 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:32.535878 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:32.538041 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:32.539282 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:32.539524 (+ 242us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:32.544980 (+ 5456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:32.544990 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:32.545010 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:32.545064 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:32.548926 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:32.550100 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:32.550109 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:32.550118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:32.550358 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:32.550970 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:32.550974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:32.550976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:32.551688 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:32.552207 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:32.552790 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 23:28:32.553276 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":463,"threads_started":1}
I20260926 23:28:32.557526 20844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c6d361f674f0bbd0700cd773d427a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260926 23:28:32.558267 20844 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3c6d361f674f0bbd0700cd773d427a; no voters:
I20260926 23:28:32.559132 20844 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:32.559284 20849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:32.560441 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:32.560729 20849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [term 1 LEADER]: Becoming Leader. State: Replica: 1c3c6d361f674f0bbd0700cd773d427a, State: Running, Role: LEADER
I20260926 23:28:32.561414 20849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c6d361f674f0bbd0700cd773d427a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } }
I20260926 23:28:32.561760 20844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:32.571911 20851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3c6d361f674f0bbd0700cd773d427a. Latest consensus state: current_term: 1 leader_uuid: "1c3c6d361f674f0bbd0700cd773d427a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c6d361f674f0bbd0700cd773d427a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } } }
I20260926 23:28:32.573087 20851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:32.574240 20862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:32.571307 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3c6d361f674f0bbd0700cd773d427a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c6d361f674f0bbd0700cd773d427a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45947 } } }
I20260926 23:28:32.575122 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:32.593083 20862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:32.604513 20862 catalog_manager.cc:1347] Generated new cluster ID: fe1b917d2c084e1abde1e8c9f7099f60
I20260926 23:28:32.604713 20862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:32.615540 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:32.574804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55533 (local address 127.0.0.1:45947)
0926 23:28:32.577024 (+ 2220us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:32.577036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:32.589219 (+ 12183us) server_negotiation.cc:407] Connection header received
0926 23:28:32.589591 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:32.589597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:32.589698 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:32.589793 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:32.592830 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:32.594108 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:32.596158 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:32.597284 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:32.597509 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:32.607626 (+ 10117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:32.607633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:32.607637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:32.607683 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:32.611558 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:32.612820 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:32.612825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:32.612828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:32.613008 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:32.613668 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:32.613673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:32.613679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:32.614032 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:32.614357 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:32.614851 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 23:28:32.615066 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2003}
I20260926 23:28:32.624037 20862 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:32.625471 20862 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:32.633327 20862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3c6d361f674f0bbd0700cd773d427a: Generated new TSK 0
I20260926 23:28:32.634339 20862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:32.642807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20777
I20260926 23:28:32.644414 20839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:32.740837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 20777
W20260926 23:28:32.763551 20762 connection.cc:535] client connection to 127.0.0.1:45947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45947): Connection reset by peer (error 104)
2026-09-26T23:28:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2170 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:32 dist-test-slave-q5g8 krb5kdc[20876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:32 dist-test-slave-q5g8 krb5kdc[20876](info): set up 2 sockets
Sep 26 23:28:32 dist-test-slave-q5g8 krb5kdc[20876](info): commencing operation
krb5kdc: starting...
W20260926 23:28:33.897225 388 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:34 dist-test-slave-q5g8 krb5kdc[20876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:34.197960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:34.581847 20889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:34.582243 20889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:34.582407 20889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:34.590781 20889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:34.590938 20889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:34.591055 20889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:34.591183 20889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:34.591274 20889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:34.602614 20889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:34.604219 20889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:34.605832 20889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:34.619390 20897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:34.619661 20894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:34.619661 20895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:34.620929 20889 server_base.cc:1034] running on GCE node
I20260926 23:28:34.621927 20889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:34.623610 20889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:34.624871 20889 hybrid_clock.cc:648] HybridClock initialized: now 1790465314624820 us; error 40 us; skew 500 ppm
I20260926 23:28:34.625245 20889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:34 dist-test-slave-q5g8 krb5kdc[20876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:34.629079 20889 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:34.630879 20889 webserver.cc:460] Webserver started at http://127.0.0.1:34811/ using document root <none> and password file <none>
I20260926 23:28:34.631703 20889 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:34.631856 20889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:34.632226 20889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:34.635836 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "fd1765e7083a4b06a5e396627c2ab4b2"
format_stamp: "Formatted at 2026-09-26 23:28:34 on dist-test-slave-q5g8"
server_key: "307f44589031322776e1935d459e48c7"
server_key_iv: "2de1295e67bffbfb85071d548ccb230e"
server_key_version: "encryptionkey@0"
I20260926 23:28:34.636827 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd1765e7083a4b06a5e396627c2ab4b2"
format_stamp: "Formatted at 2026-09-26 23:28:34 on dist-test-slave-q5g8"
server_key: "307f44589031322776e1935d459e48c7"
server_key_iv: "2de1295e67bffbfb85071d548ccb230e"
server_key_version: "encryptionkey@0"
I20260926 23:28:34.642798 20889 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:28:34.646916 20904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:34.648587 20889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:28:34.648874 20889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "fd1765e7083a4b06a5e396627c2ab4b2"
format_stamp: "Formatted at 2026-09-26 23:28:34 on dist-test-slave-q5g8"
server_key: "307f44589031322776e1935d459e48c7"
server_key_iv: "2de1295e67bffbfb85071d548ccb230e"
server_key_version: "encryptionkey@0"
I20260926 23:28:34.649121 20889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:34.680518 20889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:34.685751 20889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:34.686113 20889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:34.706218 20889 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38799
I20260926 23:28:34.706271 20955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38799 every 8 connection(s)
I20260926 23:28:34.707719 20889 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:34.708088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 20889
I20260926 23:28:34.708316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:34.708909 388 external_mini_cluster.cc:1428] Setting key 1a556e72ba1b180d5ccbb9776fb462ed
I20260926 23:28:34.712881 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:34.724067 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:34.711076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37089 (local address 127.0.0.1:38799)
0926 23:28:34.711932 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:34.711944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:34.711991 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:28:34.713860 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:34.713885 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:34.714318 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:34.714673 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:34.715346 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:34.716883 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:34.718960 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:34.719735 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:34.719893 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:34.721434 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:34.721442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:34.721458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:34.722099 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:34.722194 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:34.722226 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:34.722356 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:34.722677 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 23:28:34.722938 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":388,"threads_started":1}
I20260926 23:28:34.726855 20956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2: Bootstrap starting.
I20260926 23:28:34.728535 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:34.730808 20956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:34.732209 20956 log.cc:824] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:34.736083 20956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2: No bootstrap required, opened a new log
I20260926 23:28:34.740907 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:34.734010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34085 (local address 127.0.0.1:38799)
0926 23:28:34.734208 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:34.734217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:34.734529 (+ 312us) server_negotiation.cc:407] Connection header received
0926 23:28:34.734690 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:34.734695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:34.734775 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:34.734925 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:34.735429 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:34.736535 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:34.738002 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:34.738709 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:34.738908 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:34.739995 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:34.740002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:34.740007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:34.740031 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:34.740087 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:34.740110 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:34.740230 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:34.740418 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 23:28:34.740483 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:28:34.745719 20956 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 23:28:34.746186 20956 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:34.746326 20956 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1765e7083a4b06a5e396627c2ab4b2, State: Initialized, Role: FOLLOWER
I20260926 23:28:34.746965 20956 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 23:28:34.747229 20956 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:34.747397 20956 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:34.747614 20956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:34.750149 20956 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 23:28:34.750733 20956 leader_election.cc:302] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1765e7083a4b06a5e396627c2ab4b2; no voters:
I20260926 23:28:34.751449 20956 leader_election.cc:288] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:34.751608 20966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:34.752629 20966 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [term 1 LEADER]: Becoming Leader. State: Replica: fd1765e7083a4b06a5e396627c2ab4b2, State: Running, Role: LEADER
I20260926 23:28:34.753324 20966 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260926 23:28:34.754091 20956 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:34.756968 20968 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1765e7083a4b06a5e396627c2ab4b2. Latest consensus state: current_term: 1 leader_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260926 23:28:34.757385 20968 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:34.757903 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1765e7083a4b06a5e396627c2ab4b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260926 23:28:34.758249 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:34.758496 20973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:34.766739 20973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:34.778724 20973 catalog_manager.cc:1347] Generated new cluster ID: 299d2d979624410f91f075ce2857ffda
I20260926 23:28:34.778880 20973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:34.789707 20973 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:34.791826 20973 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:34.808943 20973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd1765e7083a4b06a5e396627c2ab4b2: Generated new TSK 0
I20260926 23:28:34.810042 20973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:34.817334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20889
W20260926 23:28:34.818953 20951 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37089
W20260926 23:28:34.819972 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 20889: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:34.820209 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 20889
W20260926 23:28:34.838811 20874 connection.cc:535] client connection to 127.0.0.1:38799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38799): Connection reset by peer (error 104)
2026-09-26T23:28:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:34 dist-test-slave-q5g8 krb5kdc[20987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:34 dist-test-slave-q5g8 krb5kdc[20987](info): set up 2 sockets
Sep 26 23:28:34 dist-test-slave-q5g8 krb5kdc[20987](info): commencing operation
krb5kdc: starting...
W20260926 23:28:35.981331 388 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:36 dist-test-slave-q5g8 krb5kdc[20987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:36.258292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:36.606737 21000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:36.607077 21000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:36.607173 21000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:36.614811 21000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:36.614943 21000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:36.614990 21000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:36.615057 21000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:36.615095 21000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:36.626631 21000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:36.628041 21000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:36.629390 21000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:36.641741 21005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:36.641770 21008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:36.641692 21006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:36.643246 21000 server_base.cc:1034] running on GCE node
I20260926 23:28:36.644196 21000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:36.645860 21000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:36.647158 21000 hybrid_clock.cc:648] HybridClock initialized: now 1790465316647094 us; error 50 us; skew 500 ppm
I20260926 23:28:36.647651 21000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:36 dist-test-slave-q5g8 krb5kdc[20987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:36.651548 21000 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:36.653502 21000 webserver.cc:460] Webserver started at http://127.0.0.1:46799/ using document root <none> and password file <none>
I20260926 23:28:36.654286 21000 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:36.654457 21000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:36.654810 21000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:36.658321 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "b6fb7b163a084139920cfe40cd0a0a63"
format_stamp: "Formatted at 2026-09-26 23:28:36 on dist-test-slave-q5g8"
server_key: "dad785d576dac76c612a70bb4deb709b"
server_key_iv: "7c8e674df94f157eddb074e485689a65"
server_key_version: "encryptionkey@0"
I20260926 23:28:36.659246 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "b6fb7b163a084139920cfe40cd0a0a63"
format_stamp: "Formatted at 2026-09-26 23:28:36 on dist-test-slave-q5g8"
server_key: "dad785d576dac76c612a70bb4deb709b"
server_key_iv: "7c8e674df94f157eddb074e485689a65"
server_key_version: "encryptionkey@0"
I20260926 23:28:36.665316 21000 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:28:36.669296 21015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:36.670720 21000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:28:36.670925 21000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "b6fb7b163a084139920cfe40cd0a0a63"
format_stamp: "Formatted at 2026-09-26 23:28:36 on dist-test-slave-q5g8"
server_key: "dad785d576dac76c612a70bb4deb709b"
server_key_iv: "7c8e674df94f157eddb074e485689a65"
server_key_version: "encryptionkey@0"
I20260926 23:28:36.671149 21000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:36.677534 21000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:36.682133 21000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:36.682466 21000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:36.702524 21000 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41771
I20260926 23:28:36.702570 21066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41771 every 8 connection(s)
I20260926 23:28:36.704094 21000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:36.704638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21000
I20260926 23:28:36.704872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:36.705518 388 external_mini_cluster.cc:1428] Setting key f0fdafff5cf0ed464b005a9167c15ab1
I20260926 23:28:36.709162 21067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:36.714457 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:36.707502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45835 (local address 127.0.0.1:41771)
0926 23:28:36.708410 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:36.708426 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:36.708470 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:28:36.710447 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:36.710472 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:36.710864 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:36.711223 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:36.712539 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:36.712547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:36.712562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:36.712582 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:36.712741 (+ 159us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:36.712781 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:36.713006 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:36.713141 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 23:28:36.713376 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":383,"threads_started":1}
I20260926 23:28:36.718745 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:36.723837 21067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63: Bootstrap starting.
I20260926 23:28:36.727164 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:36.724223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44759 (local address 127.0.0.1:41771)
0926 23:28:36.724488 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:36.724498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:36.725100 (+ 602us) server_negotiation.cc:407] Connection header received
0926 23:28:36.725350 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:36.725359 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:36.725411 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:36.725534 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:36.726496 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:36.726502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:36.726507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:36.726525 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:36.726576 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:36.726591 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:36.726714 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:36.726809 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 23:28:36.726881 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:28:36.727458 21067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:36.728808 21067 log.cc:824] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:36.731765 21067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63: No bootstrap required, opened a new log
I20260926 23:28:36.739920 21067 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fb7b163a084139920cfe40cd0a0a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
I20260926 23:28:36.740348 21067 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:36.740451 21067 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fb7b163a084139920cfe40cd0a0a63, State: Initialized, Role: FOLLOWER
I20260926 23:28:36.741178 21067 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fb7b163a084139920cfe40cd0a0a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
I20260926 23:28:36.741426 21067 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:36.741600 21067 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:36.741796 21067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:36.744316 21067 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fb7b163a084139920cfe40cd0a0a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
I20260926 23:28:36.744889 21067 leader_election.cc:302] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6fb7b163a084139920cfe40cd0a0a63; no voters:
I20260926 23:28:36.745536 21067 leader_election.cc:288] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:36.745674 21077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:36.746379 21077 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [term 1 LEADER]: Becoming Leader. State: Replica: b6fb7b163a084139920cfe40cd0a0a63, State: Running, Role: LEADER
I20260926 23:28:36.746924 21077 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fb7b163a084139920cfe40cd0a0a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } }
I20260926 23:28:36.747668 21067 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:36.750088 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fb7b163a084139920cfe40cd0a0a63. Latest consensus state: current_term: 1 leader_uuid: "b6fb7b163a084139920cfe40cd0a0a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fb7b163a084139920cfe40cd0a0a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } } }
I20260926 23:28:36.750588 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:36.750730 21078 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6fb7b163a084139920cfe40cd0a0a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fb7b163a084139920cfe40cd0a0a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41771 } } }
I20260926 23:28:36.752103 21078 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:36.754000 21086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:36.758443 21086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:36.768072 21086 catalog_manager.cc:1347] Generated new cluster ID: d60d32ba90d5453b8ce64a2944cd07c5
I20260926 23:28:36.768252 21086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:36.794373 21086 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:36.794907 21086 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:36.804076 21086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6fb7b163a084139920cfe40cd0a0a63: Generated new TSK 0
I20260926 23:28:36.805141 21086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:36.809263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21000
W20260926 23:28:36.810580 21062 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45835
W20260926 23:28:36.811323 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21000: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:36.811506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21000
2026-09-26T23:28:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:36 dist-test-slave-q5g8 krb5kdc[21098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:36 dist-test-slave-q5g8 krb5kdc[21098](info): set up 2 sockets
Sep 26 23:28:36 dist-test-slave-q5g8 krb5kdc[21098](info): commencing operation
krb5kdc: starting...
W20260926 23:28:37.921092 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:38 dist-test-slave-q5g8 krb5kdc[21098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:38.185110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:38.535559 21111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:38.535902 21111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:38.536069 21111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:38.543680 21111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:38.543809 21111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:38.543916 21111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:38.544023 21111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:38.544106 21111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:38.554425 21111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:38.555828 21111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:38.557269 21111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:38.569474 21116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:38.569506 21117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:38.569551 21119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:38.570477 21111 server_base.cc:1034] running on GCE node
I20260926 23:28:38.571255 21111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:38.572649 21111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:38.573911 21111 hybrid_clock.cc:648] HybridClock initialized: now 1790465318573850 us; error 56 us; skew 500 ppm
I20260926 23:28:38.574316 21111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:38 dist-test-slave-q5g8 krb5kdc[21098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:38.584749 21111 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:38.586678 21111 webserver.cc:460] Webserver started at http://127.0.0.1:36915/ using document root <none> and password file <none>
I20260926 23:28:38.587507 21111 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:38.587670 21111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:38.588068 21111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:38.591532 21111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "7ff6401101e74030b6287359f9c75264"
format_stamp: "Formatted at 2026-09-26 23:28:38 on dist-test-slave-q5g8"
server_key: "22f50425d84e0fc33d17f9d10937c442"
server_key_iv: "f93986becfea9d2545a69253b6de459c"
server_key_version: "encryptionkey@0"
I20260926 23:28:38.592381 21111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "7ff6401101e74030b6287359f9c75264"
format_stamp: "Formatted at 2026-09-26 23:28:38 on dist-test-slave-q5g8"
server_key: "22f50425d84e0fc33d17f9d10937c442"
server_key_iv: "f93986becfea9d2545a69253b6de459c"
server_key_version: "encryptionkey@0"
I20260926 23:28:38.597936 21111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 23:28:38.601687 21126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:38.603098 21111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:28:38.603291 21111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "7ff6401101e74030b6287359f9c75264"
format_stamp: "Formatted at 2026-09-26 23:28:38 on dist-test-slave-q5g8"
server_key: "22f50425d84e0fc33d17f9d10937c442"
server_key_iv: "f93986becfea9d2545a69253b6de459c"
server_key_version: "encryptionkey@0"
I20260926 23:28:38.603487 21111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:38.627573 21111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:38.632390 21111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:38.632737 21111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:38.652773 21111 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41695
I20260926 23:28:38.652778 21177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41695 every 8 connection(s)
I20260926 23:28:38.654275 21111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:38.658290 21178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:38.661968 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21111
I20260926 23:28:38.662165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:38.662720 388 external_mini_cluster.cc:1428] Setting key 08df2e0ff26425e9173dd3fb231dee68
I20260926 23:28:38.670679 21178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264: Bootstrap starting.
I20260926 23:28:38.674013 21178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:38.675051 21178 log.cc:824] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:28:38 dist-test-slave-q5g8 krb5kdc[21098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:28:38.678048 21178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264: No bootstrap required, opened a new log
I20260926 23:28:38.685280 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:38.664605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52207 (local address 127.0.0.1:41695)
0926 23:28:38.665413 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:38.665425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:38.665466 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:28:38.667157 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:38.667179 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:38.667643 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:38.668075 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:38.669639 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:38.671010 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:38.672308 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:38.673059 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:38.673265 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:38.677435 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:38.677443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:38.677460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:38.677498 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:38.680799 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:38.681581 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:38.681593 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:38.681599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:38.681776 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:38.682188 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:38.682192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:38.682198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:38.682791 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:38.683124 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:38.683593 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 23:28:38.684116 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":346,"threads_started":1}
I20260926 23:28:38.686619 21178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6401101e74030b6287359f9c75264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41695 } }
I20260926 23:28:38.687083 21178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:38.687201 21178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff6401101e74030b6287359f9c75264, State: Initialized, Role: FOLLOWER
I20260926 23:28:38.687796 21178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6401101e74030b6287359f9c75264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41695 } }
I20260926 23:28:38.688112 21178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:38.688295 21178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:38.688473 21178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:38.689576 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:38.690972 21178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6401101e74030b6287359f9c75264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41695 } }
I20260926 23:28:38.691471 21178 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff6401101e74030b6287359f9c75264; no voters:
I20260926 23:28:38.692186 21178 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:38.692304 21183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:38.694023 21183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff6401101e74030b6287359f9c75264, State: Running, Role: LEADER
I20260926 23:28:38.694725 21183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6401101e74030b6287359f9c75264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41695 } }
I20260926 23:28:38.694947 21178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:38.700940 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff6401101e74030b6287359f9c75264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6401101e74030b6287359f9c75264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41695 } } }
I20260926 23:28:38.701502 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:38.702138 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff6401101e74030b6287359f9c75264. Latest consensus state: current_term: 1 leader_uuid: "7ff6401101e74030b6287359f9c75264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6401101e74030b6287359f9c75264" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41695 } } }
I20260926 23:28:38.702490 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:38.712855 21196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:38.720489 21196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:38.729175 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:38.695059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39609 (local address 127.0.0.1:41695)
0926 23:28:38.695224 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:38.695228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:38.710273 (+ 15045us) server_negotiation.cc:407] Connection header received
0926 23:28:38.711597 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:38.711610 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:38.711731 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:38.711826 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:38.713961 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:38.714771 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:38.716483 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:38.717225 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:38.717366 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:38.721020 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:38.721028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:38.721034 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:38.721082 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:38.725393 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:38.726422 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:38.726432 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:38.726436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:38.726621 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:38.727129 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:38.727134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:38.727138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:38.727637 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:38.727887 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:38.728515 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 23:28:38.728786 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 23:28:38.739698 21196 catalog_manager.cc:1347] Generated new cluster ID: d538b4d9364747c59abb1474e8d2ac3f
I20260926 23:28:38.739856 21196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:38.764283 21196 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:38.765955 21196 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:38.782951 21196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ff6401101e74030b6287359f9c75264: Generated new TSK 0
I20260926 23:28:38.783859 21196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:38.797602 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21111
I20260926 23:28:38.798877 21173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:38.894371 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21111
2026-09-26T23:28:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2083 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:38 dist-test-slave-q5g8 krb5kdc[21211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:38 dist-test-slave-q5g8 krb5kdc[21211](info): set up 2 sockets
Sep 26 23:28:38 dist-test-slave-q5g8 krb5kdc[21211](info): commencing operation
krb5kdc: starting...
W20260926 23:28:40.125090 388 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:40 dist-test-slave-q5g8 krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:40.401646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:40.743090 21224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:40.743418 21224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:40.743599 21224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:40.751124 21224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:40.751251 21224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:40.751367 21224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:40.751469 21224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:40.751534 21224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:40.761899 21224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:40.763315 21224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:40.764700 21224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:40.776850 21232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:40.776815 21230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:40.776787 21229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:40.778148 21224 server_base.cc:1034] running on GCE node
I20260926 23:28:40.779003 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:40.780634 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:40.781898 21224 hybrid_clock.cc:648] HybridClock initialized: now 1790465320781838 us; error 57 us; skew 500 ppm
I20260926 23:28:40.782256 21224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:40 dist-test-slave-q5g8 krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:40.785434 21224 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:40.787050 21224 webserver.cc:460] Webserver started at http://127.0.0.1:42983/ using document root <none> and password file <none>
I20260926 23:28:40.787742 21224 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:40.787892 21224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:40.788205 21224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:40.791254 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "f0c362e5d13f48a2a01611f258a362ad"
format_stamp: "Formatted at 2026-09-26 23:28:40 on dist-test-slave-q5g8"
server_key: "47a4683d7d27723c8e8dbef8368bbb5c"
server_key_iv: "c68cfebbceec2b91bc071b9f073d9bb6"
server_key_version: "encryptionkey@0"
I20260926 23:28:40.792053 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0c362e5d13f48a2a01611f258a362ad"
format_stamp: "Formatted at 2026-09-26 23:28:40 on dist-test-slave-q5g8"
server_key: "47a4683d7d27723c8e8dbef8368bbb5c"
server_key_iv: "c68cfebbceec2b91bc071b9f073d9bb6"
server_key_version: "encryptionkey@0"
I20260926 23:28:40.797482 21224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 23:28:40.801116 21239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:40.802492 21224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:28:40.802699 21224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "f0c362e5d13f48a2a01611f258a362ad"
format_stamp: "Formatted at 2026-09-26 23:28:40 on dist-test-slave-q5g8"
server_key: "47a4683d7d27723c8e8dbef8368bbb5c"
server_key_iv: "c68cfebbceec2b91bc071b9f073d9bb6"
server_key_version: "encryptionkey@0"
I20260926 23:28:40.802915 21224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:40.822772 21224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:40.827090 21224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:40.827394 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:40.846962 21224 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38587
I20260926 23:28:40.847002 21290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38587 every 8 connection(s)
I20260926 23:28:40.848444 21224 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:40.852375 21291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:40.858757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21224
I20260926 23:28:40.858970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:40.859500 388 external_mini_cluster.cc:1428] Setting key 6d8e4217570d5816a4a794d21ca19176
I20260926 23:28:40.864007 21291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad: Bootstrap starting.
I20260926 23:28:40.867671 21291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:40.869035 21291 log.cc:824] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:40.872264 21291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad: No bootstrap required, opened a new log
I20260926 23:28:40.873801 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:40.861533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35575 (local address 127.0.0.1:38587)
0926 23:28:40.862440 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:40.862452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:40.862494 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:28:40.864265 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:40.864287 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:40.864760 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:40.865108 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:40.865746 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:40.867047 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:40.869063 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:40.869865 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:40.870081 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:40.871247 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:40.871255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:40.871278 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:40.871846 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:40.871929 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:40.871955 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:40.872091 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:40.872334 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 23:28:40.872820 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":473,"threads_started":1}
I20260926 23:28:40.877835 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:40.881124 21291 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c362e5d13f48a2a01611f258a362ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 23:28:40.881562 21291 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:40.881682 21291 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c362e5d13f48a2a01611f258a362ad, State: Initialized, Role: FOLLOWER
I20260926 23:28:40.882354 21291 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c362e5d13f48a2a01611f258a362ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 23:28:40.882678 21291 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:40.882922 21291 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:40.883116 21291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:40.886219 21291 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c362e5d13f48a2a01611f258a362ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 23:28:40.886801 21291 leader_election.cc:302] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0c362e5d13f48a2a01611f258a362ad; no voters:
I20260926 23:28:40.887530 21291 leader_election.cc:288] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:40.887646 21301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:40.888495 21301 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [term 1 LEADER]: Becoming Leader. State: Replica: f0c362e5d13f48a2a01611f258a362ad, State: Running, Role: LEADER
I20260926 23:28:40.889240 21301 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c362e5d13f48a2a01611f258a362ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 23:28:40.889947 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:40.882628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55917 (local address 127.0.0.1:38587)
0926 23:28:40.882848 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:40.882856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:40.883286 (+ 430us) server_negotiation.cc:407] Connection header received
0926 23:28:40.883542 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:40.883547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:40.883632 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:40.883771 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:40.884211 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:40.885318 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:40.886706 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:40.887429 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:40.887577 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:40.888849 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:40.888856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:40.888860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:40.888884 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:40.888928 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:40.888947 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:40.889034 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:40.889534 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 23:28:40.889626 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:28:40.890337 21291 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:40.892454 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0c362e5d13f48a2a01611f258a362ad. Latest consensus state: current_term: 1 leader_uuid: "f0c362e5d13f48a2a01611f258a362ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c362e5d13f48a2a01611f258a362ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } } }
I20260926 23:28:40.892859 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:40.893532 21302 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0c362e5d13f48a2a01611f258a362ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c362e5d13f48a2a01611f258a362ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } } }
I20260926 23:28:40.893854 21302 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:40.895563 21306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:40.901708 21306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:40.911419 21306 catalog_manager.cc:1347] Generated new cluster ID: 78326efc74c24ba2a766bee742148b2c
I20260926 23:28:40.911567 21306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:40.928049 21306 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:40.930069 21306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:40.946153 21306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0c362e5d13f48a2a01611f258a362ad: Generated new TSK 0
I20260926 23:28:40.947183 21306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:40.960691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21224
W20260926 23:28:40.961831 21286 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35575
W20260926 23:28:40.962561 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21224: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:40.962729 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21224
2026-09-26T23:28:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:41 dist-test-slave-q5g8 krb5kdc[21322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:41 dist-test-slave-q5g8 krb5kdc[21322](info): set up 2 sockets
Sep 26 23:28:41 dist-test-slave-q5g8 krb5kdc[21322](info): commencing operation
krb5kdc: starting...
W20260926 23:28:42.097370 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:42 dist-test-slave-q5g8 krb5kdc[21322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:42.384083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:42.739826 21335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:42.740139 21335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:42.740322 21335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:42.747968 21335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:42.748095 21335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:42.748183 21335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:42.748306 21335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:42.748368 21335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:42.758558 21335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:42.759960 21335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:42.761472 21335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:42.777160 21340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:42.777115 21343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:42.777201 21341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:42.778475 21335 server_base.cc:1034] running on GCE node
I20260926 23:28:42.779280 21335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:42.780817 21335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:42.782064 21335 hybrid_clock.cc:648] HybridClock initialized: now 1790465322782016 us; error 56 us; skew 500 ppm
I20260926 23:28:42.782434 21335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:42 dist-test-slave-q5g8 krb5kdc[21322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:42.785938 21335 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:42.787496 21335 webserver.cc:460] Webserver started at http://127.0.0.1:35339/ using document root <none> and password file <none>
I20260926 23:28:42.788245 21335 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:42.788399 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:42.788842 21335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:42.791977 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "2f228b2f6a1f4a4aadffc43069354497"
format_stamp: "Formatted at 2026-09-26 23:28:42 on dist-test-slave-q5g8"
server_key: "fd28944c4b6e5601b220a6185fbc4696"
server_key_iv: "ef0dc5861ec3159cd82f05151630bc4e"
server_key_version: "encryptionkey@0"
I20260926 23:28:42.792876 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f228b2f6a1f4a4aadffc43069354497"
format_stamp: "Formatted at 2026-09-26 23:28:42 on dist-test-slave-q5g8"
server_key: "fd28944c4b6e5601b220a6185fbc4696"
server_key_iv: "ef0dc5861ec3159cd82f05151630bc4e"
server_key_version: "encryptionkey@0"
I20260926 23:28:42.798485 21335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 23:28:42.802253 21350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:42.803836 21335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:28:42.804064 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "2f228b2f6a1f4a4aadffc43069354497"
format_stamp: "Formatted at 2026-09-26 23:28:42 on dist-test-slave-q5g8"
server_key: "fd28944c4b6e5601b220a6185fbc4696"
server_key_iv: "ef0dc5861ec3159cd82f05151630bc4e"
server_key_version: "encryptionkey@0"
I20260926 23:28:42.804307 21335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:42.810755 21335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:42.816008 21335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:42.816366 21335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:42.835976 21335 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40295
I20260926 23:28:42.836017 21401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40295 every 8 connection(s)
I20260926 23:28:42.837572 21335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:42.841176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21335
I20260926 23:28:42.841395 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:42.841617 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:42.841955 388 external_mini_cluster.cc:1428] Setting key d702be6661447c2b980a8c3275966cbc
I20260926 23:28:42.850271 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:42.843786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47663 (local address 127.0.0.1:40295)
0926 23:28:42.844664 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:42.844676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:42.844775 (+ 99us) server_negotiation.cc:407] Connection header received
0926 23:28:42.846510 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:42.846534 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:42.846925 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:42.847306 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:42.848418 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:42.848427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:42.848441 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:42.848458 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:42.848521 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:42.848541 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:42.848865 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:42.848975 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 23:28:42.849237 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":396,"threads_started":1}
I20260926 23:28:42.854732 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:42.856837 21402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497: Bootstrap starting.
I20260926 23:28:42.861469 21402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:42.862700 21402 log.cc:824] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:42.863441 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:42.860595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52585 (local address 127.0.0.1:40295)
0926 23:28:42.860828 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:42.860834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:42.861264 (+ 430us) server_negotiation.cc:407] Connection header received
0926 23:28:42.861418 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:42.861425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:42.861505 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:42.861624 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:42.862609 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:42.862615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:42.862619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:42.862640 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:42.862702 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:42.862725 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:42.862883 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:42.863025 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 23:28:42.863082 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:28:42.865912 21402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497: No bootstrap required, opened a new log
I20260926 23:28:42.873317 21402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f228b2f6a1f4a4aadffc43069354497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 23:28:42.873744 21402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:42.873886 21402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f228b2f6a1f4a4aadffc43069354497, State: Initialized, Role: FOLLOWER
I20260926 23:28:42.874577 21402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f228b2f6a1f4a4aadffc43069354497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 23:28:42.874797 21402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:42.874903 21402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:42.875043 21402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:42.877331 21402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f228b2f6a1f4a4aadffc43069354497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 23:28:42.877830 21402 leader_election.cc:302] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f228b2f6a1f4a4aadffc43069354497; no voters:
I20260926 23:28:42.878438 21402 leader_election.cc:288] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:42.878605 21412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:42.879289 21412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [term 1 LEADER]: Becoming Leader. State: Replica: 2f228b2f6a1f4a4aadffc43069354497, State: Running, Role: LEADER
I20260926 23:28:42.879814 21412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f228b2f6a1f4a4aadffc43069354497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 23:28:42.880584 21402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:42.882866 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f228b2f6a1f4a4aadffc43069354497. Latest consensus state: current_term: 1 leader_uuid: "2f228b2f6a1f4a4aadffc43069354497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f228b2f6a1f4a4aadffc43069354497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } } }
I20260926 23:28:42.884043 21413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f228b2f6a1f4a4aadffc43069354497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f228b2f6a1f4a4aadffc43069354497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } } }
I20260926 23:28:42.884428 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:42.884455 21413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:42.886080 21422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:42.891247 21422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:42.900775 21422 catalog_manager.cc:1347] Generated new cluster ID: e3cd9d13f54840898c055ae510ff55d2
I20260926 23:28:42.900906 21422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:42.935197 21422 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:42.935745 21422 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:42.943972 21422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f228b2f6a1f4a4aadffc43069354497: Generated new TSK 0
I20260926 23:28:42.945098 21422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:42.960304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21335
W20260926 23:28:42.961815 21397 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47663
W20260926 23:28:42.962677 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21335: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:42.962854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21335
2026-09-26T23:28:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:43 dist-test-slave-q5g8 krb5kdc[21433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:43 dist-test-slave-q5g8 krb5kdc[21433](info): set up 2 sockets
Sep 26 23:28:43 dist-test-slave-q5g8 krb5kdc[21433](info): commencing operation
krb5kdc: starting...
W20260926 23:28:44.109766 388 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:44 dist-test-slave-q5g8 krb5kdc[21433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:44.405838 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:44.774868 21446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:44.775297 21446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:44.775494 21446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:44.783335 21446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:44.783540 21446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:44.783669 21446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:44.783785 21446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:44.783877 21446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:44.795061 21446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:44.796654 21446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:44.798184 21446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:44.811061 21451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:44.811085 21452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:44.811023 21454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:44.811872 21446 server_base.cc:1034] running on GCE node
I20260926 23:28:44.812884 21446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:44.814482 21446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:44.815739 21446 hybrid_clock.cc:648] HybridClock initialized: now 1790465324815664 us; error 39 us; skew 500 ppm
I20260926 23:28:44.816148 21446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:44 dist-test-slave-q5g8 krb5kdc[21433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:44.834424 21446 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:44.836318 21446 webserver.cc:460] Webserver started at http://127.0.0.1:42585/ using document root <none> and password file <none>
I20260926 23:28:44.837240 21446 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:44.837404 21446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:44.837740 21446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:44.841117 21446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "cd18336903d245f1b1b29c0b5c1c415e"
format_stamp: "Formatted at 2026-09-26 23:28:44 on dist-test-slave-q5g8"
server_key: "0ca5dea56f90be1c7913dd30c71ddb12"
server_key_iv: "aab13c8e5d865b2a0fd5211290bb49fa"
server_key_version: "encryptionkey@0"
I20260926 23:28:44.842018 21446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "cd18336903d245f1b1b29c0b5c1c415e"
format_stamp: "Formatted at 2026-09-26 23:28:44 on dist-test-slave-q5g8"
server_key: "0ca5dea56f90be1c7913dd30c71ddb12"
server_key_iv: "aab13c8e5d865b2a0fd5211290bb49fa"
server_key_version: "encryptionkey@0"
I20260926 23:28:44.847836 21446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:28:44.851920 21461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:44.853482 21446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 23:28:44.853719 21446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "cd18336903d245f1b1b29c0b5c1c415e"
format_stamp: "Formatted at 2026-09-26 23:28:44 on dist-test-slave-q5g8"
server_key: "0ca5dea56f90be1c7913dd30c71ddb12"
server_key_iv: "aab13c8e5d865b2a0fd5211290bb49fa"
server_key_version: "encryptionkey@0"
I20260926 23:28:44.853946 21446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:44.876006 21446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:44.880746 21446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:44.881078 21446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:44.902097 21446 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44455
I20260926 23:28:44.902150 21512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44455 every 8 connection(s)
I20260926 23:28:44.903719 21446 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:44.904841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21446
I20260926 23:28:44.905056 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:44.905629 388 external_mini_cluster.cc:1428] Setting key 268ff48f45ba94365339f71aed37f138
I20260926 23:28:44.909358 21513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:28:44 dist-test-slave-q5g8 krb5kdc[21433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:28:44.925966 21513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e: Bootstrap starting.
I20260926 23:28:44.930135 21513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:44.931784 21513 log.cc:824] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:44.935163 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:44.907909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:43777 (local address 127.0.0.1:44455)
0926 23:28:44.908884 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:44.908898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:44.908947 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:28:44.910967 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:44.910994 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:44.911485 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:44.911879 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:44.914331 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:44.915916 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:44.918140 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:44.919056 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:44.919484 (+ 428us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:44.925471 (+ 5987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:44.925480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:44.925507 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:44.925557 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:44.929056 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:44.930180 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:44.930193 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:44.930203 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:44.930466 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:44.931088 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:44.931094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:44.931098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:44.932021 (+ 923us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:44.932441 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:44.933007 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 23:28:44.933505 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":434,"threads_started":1}
I20260926 23:28:44.936131 21513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e: No bootstrap required, opened a new log
I20260926 23:28:44.940320 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:44.947024 21513 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd18336903d245f1b1b29c0b5c1c415e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260926 23:28:44.947566 21513 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:44.947705 21513 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd18336903d245f1b1b29c0b5c1c415e, State: Initialized, Role: FOLLOWER
I20260926 23:28:44.948626 21513 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd18336903d245f1b1b29c0b5c1c415e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260926 23:28:44.949028 21513 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:44.949222 21513 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:44.949486 21513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:44.952957 21513 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd18336903d245f1b1b29c0b5c1c415e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260926 23:28:44.953707 21513 leader_election.cc:302] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd18336903d245f1b1b29c0b5c1c415e; no voters:
I20260926 23:28:44.954471 21513 leader_election.cc:288] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:44.954666 21523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:44.957283 21513 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:44.958678 21523 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [term 1 LEADER]: Becoming Leader. State: Replica: cd18336903d245f1b1b29c0b5c1c415e, State: Running, Role: LEADER
I20260926 23:28:44.959569 21523 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd18336903d245f1b1b29c0b5c1c415e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260926 23:28:44.962289 21525 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd18336903d245f1b1b29c0b5c1c415e. Latest consensus state: current_term: 1 leader_uuid: "cd18336903d245f1b1b29c0b5c1c415e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd18336903d245f1b1b29c0b5c1c415e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } } }
I20260926 23:28:44.962877 21525 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:44.963399 21524 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd18336903d245f1b1b29c0b5c1c415e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd18336903d245f1b1b29c0b5c1c415e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } } }
I20260926 23:28:44.963882 21524 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:44.966410 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:44.945127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:34627 (local address 127.0.0.1:44455)
0926 23:28:44.945325 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:44.945329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:44.945863 (+ 534us) server_negotiation.cc:407] Connection header received
0926 23:28:44.946025 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:44.946032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:44.946149 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:44.946282 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:44.948053 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:44.949493 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:44.951111 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:44.951912 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:44.952069 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:44.955548 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:44.955559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:44.955568 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:44.955636 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:44.960579 (+ 4943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:44.961997 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:44.962002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:44.962005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:44.962265 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:44.962966 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:44.962974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:44.962979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:44.963488 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:44.963961 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:44.965568 (+ 1607us) server_negotiation.cc:299] Negotiation successful
0926 23:28:44.965852 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:28:44.971292 21530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:44.978376 21530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:44.989073 21530 catalog_manager.cc:1347] Generated new cluster ID: 6443fb34aef84175a20ef573fe9d57fb
I20260926 23:28:44.989243 21530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:45.013432 21530 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:45.015117 21530 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:45.028491 21530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd18336903d245f1b1b29c0b5c1c415e: Generated new TSK 0
I20260926 23:28:45.029841 21530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:45.045591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21446
I20260926 23:28:45.047739 21508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:45.146795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21446
W20260926 23:28:45.165853 21431 connection.cc:535] client connection to 127.0.0.1:44455 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44455): Connection reset by peer (error 104)
2026-09-26T23:28:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:45 dist-test-slave-q5g8 krb5kdc[21545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:45 dist-test-slave-q5g8 krb5kdc[21545](info): set up 2 sockets
Sep 26 23:28:45 dist-test-slave-q5g8 krb5kdc[21545](info): commencing operation
krb5kdc: starting...
W20260926 23:28:46.305406 388 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:46 dist-test-slave-q5g8 krb5kdc[21545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:46.597481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:46.949301 21558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:46.949682 21558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:46.949859 21558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:46.957350 21558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:46.957554 21558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:46.957669 21558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:46.957787 21558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:46.957875 21558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:46.969825 21558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:46.971429 21558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:46.972992 21558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:46.986780 21563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:46.986811 21564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:46.987080 21566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:46.987888 21558 server_base.cc:1034] running on GCE node
I20260926 23:28:46.988958 21558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:46.990688 21558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:46.991973 21558 hybrid_clock.cc:648] HybridClock initialized: now 1790465326991915 us; error 59 us; skew 500 ppm
I20260926 23:28:46.992417 21558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:46 dist-test-slave-q5g8 krb5kdc[21545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:46.996223 21558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:46.998294 21558 webserver.cc:460] Webserver started at http://127.0.0.1:41069/ using document root <none> and password file <none>
I20260926 23:28:46.999121 21558 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:46.999295 21558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:46.999650 21558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:47.003086 21558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "0e7f9e27c1a74e0ea56fd0856e54c227"
format_stamp: "Formatted at 2026-09-26 23:28:46 on dist-test-slave-q5g8"
server_key: "54e776fe91423d21466f9b65b15bb8b9"
server_key_iv: "58f15ecef90182e508570e092083bdea"
server_key_version: "encryptionkey@0"
I20260926 23:28:47.003999 21558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e7f9e27c1a74e0ea56fd0856e54c227"
format_stamp: "Formatted at 2026-09-26 23:28:46 on dist-test-slave-q5g8"
server_key: "54e776fe91423d21466f9b65b15bb8b9"
server_key_iv: "58f15ecef90182e508570e092083bdea"
server_key_version: "encryptionkey@0"
I20260926 23:28:47.010385 21558 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:28:47.015036 21573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:47.016956 21558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:28:47.017197 21558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "0e7f9e27c1a74e0ea56fd0856e54c227"
format_stamp: "Formatted at 2026-09-26 23:28:46 on dist-test-slave-q5g8"
server_key: "54e776fe91423d21466f9b65b15bb8b9"
server_key_iv: "58f15ecef90182e508570e092083bdea"
server_key_version: "encryptionkey@0"
I20260926 23:28:47.017454 21558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:47.032946 21558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:47.038393 21558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:47.038761 21558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:47.060520 21558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41829
I20260926 23:28:47.060559 21624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41829 every 8 connection(s)
I20260926 23:28:47.062306 21558 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:47.065744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21558
I20260926 23:28:47.065953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:47.066424 388 external_mini_cluster.cc:1428] Setting key 7ecd5cd4bb68170b6c45b14f9b719293
I20260926 23:28:47.067062 21625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:47.084379 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:47.068565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:47267 (local address 127.0.0.1:41829)
0926 23:28:47.069527 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:47.069545 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:47.069602 (+ 57us) server_negotiation.cc:407] Connection header received
0926 23:28:47.072548 (+ 2946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:47.072589 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:47.073349 (+ 760us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:47.073928 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:47.074593 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:47.076929 (+ 2336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:47.078500 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:47.079687 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:47.079919 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:47.081225 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:47.081242 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:47.081270 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:47.082121 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:47.082236 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:47.082270 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:47.082445 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:47.082710 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 23:28:47.082934 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":374,"threads_started":1}
I20260926 23:28:47.084334 21625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227: Bootstrap starting.
I20260926 23:28:47.088183 21625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:47.088915 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:47.089560 21625 log.cc:824] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:47.093094 21625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227: No bootstrap required, opened a new log
I20260926 23:28:47.102319 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:47.094261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:35405 (local address 127.0.0.1:41829)
0926 23:28:47.094517 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:47.094527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:47.095077 (+ 550us) server_negotiation.cc:407] Connection header received
0926 23:28:47.095327 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:47.095332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:47.095416 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:47.095511 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:47.096237 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:47.097194 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:47.099286 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:47.099798 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:47.099907 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:47.101440 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:47.101445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:47.101448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:47.101469 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:47.101513 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:47.101529 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:47.101620 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:47.101848 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 23:28:47.101901 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:28:47.104116 21625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41829 } }
I20260926 23:28:47.104609 21625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:47.104801 21625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7f9e27c1a74e0ea56fd0856e54c227, State: Initialized, Role: FOLLOWER
I20260926 23:28:47.105459 21625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41829 } }
I20260926 23:28:47.105726 21625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:47.105890 21625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:47.106081 21625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:47.108435 21625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41829 } }
I20260926 23:28:47.109051 21625 leader_election.cc:302] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e7f9e27c1a74e0ea56fd0856e54c227; no voters:
I20260926 23:28:47.109745 21625 leader_election.cc:288] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:47.109905 21635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:47.110670 21635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [term 1 LEADER]: Becoming Leader. State: Replica: 0e7f9e27c1a74e0ea56fd0856e54c227, State: Running, Role: LEADER
I20260926 23:28:47.111311 21635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41829 } }
I20260926 23:28:47.112109 21625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:47.114902 21636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41829 } } }
I20260926 23:28:47.115135 21637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e7f9e27c1a74e0ea56fd0856e54c227. Latest consensus state: current_term: 1 leader_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f9e27c1a74e0ea56fd0856e54c227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41829 } } }
I20260926 23:28:47.115449 21637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:47.116958 21636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:47.118558 21642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:47.124310 21642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:47.134428 21642 catalog_manager.cc:1347] Generated new cluster ID: f5f8ea6fe48e4941b4276b1ffc35f5be
I20260926 23:28:47.134558 21642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:47.144920 21642 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:47.146394 21642 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:47.159538 21642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e7f9e27c1a74e0ea56fd0856e54c227: Generated new TSK 0
I20260926 23:28:47.160888 21642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:47.169596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21558
W20260926 23:28:47.171058 21620 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:47267
W20260926 23:28:47.171916 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21558: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:47.172091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21558
W20260926 23:28:47.189687 21543 connection.cc:535] client connection to 127.0.0.1:41829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41829): Connection reset by peer (error 104)
2026-09-26T23:28:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:47 dist-test-slave-q5g8 krb5kdc[21656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:47 dist-test-slave-q5g8 krb5kdc[21656](info): set up 2 sockets
Sep 26 23:28:47 dist-test-slave-q5g8 krb5kdc[21656](info): commencing operation
krb5kdc: starting...
W20260926 23:28:48.337340 388 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:48 dist-test-slave-q5g8 krb5kdc[21656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:48.625072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:48.987327 21669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:48.987664 21669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:48.987848 21669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:48.995661 21669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:48.995813 21669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:48.995914 21669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:48.996012 21669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:48.996099 21669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:49.007222 21669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:49.008767 21669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:49.010321 21669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:49.023980 21674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:49.023980 21675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:49.024014 21677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:49.025122 21669 server_base.cc:1034] running on GCE node
I20260926 23:28:49.026129 21669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:49.027824 21669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:49.029067 21669 hybrid_clock.cc:648] HybridClock initialized: now 1790465329029030 us; error 67 us; skew 500 ppm
I20260926 23:28:49.029466 21669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:49 dist-test-slave-q5g8 krb5kdc[21656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:49.033102 21669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:49.034868 21669 webserver.cc:460] Webserver started at http://127.0.0.1:35169/ using document root <none> and password file <none>
I20260926 23:28:49.035646 21669 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:49.035818 21669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:49.036170 21669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:49.039557 21669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "9a5a84248cd647b9865b6da57aadc032"
format_stamp: "Formatted at 2026-09-26 23:28:49 on dist-test-slave-q5g8"
server_key: "ba1dda97ecd650c1b109864271535767"
server_key_iv: "f99c2bc972e55253865aab7f3848f4ab"
server_key_version: "encryptionkey@0"
I20260926 23:28:49.040459 21669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a5a84248cd647b9865b6da57aadc032"
format_stamp: "Formatted at 2026-09-26 23:28:49 on dist-test-slave-q5g8"
server_key: "ba1dda97ecd650c1b109864271535767"
server_key_iv: "f99c2bc972e55253865aab7f3848f4ab"
server_key_version: "encryptionkey@0"
I20260926 23:28:49.046720 21669 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:28:49.050853 21684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:49.052409 21669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:28:49.052639 21669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "9a5a84248cd647b9865b6da57aadc032"
format_stamp: "Formatted at 2026-09-26 23:28:49 on dist-test-slave-q5g8"
server_key: "ba1dda97ecd650c1b109864271535767"
server_key_iv: "f99c2bc972e55253865aab7f3848f4ab"
server_key_version: "encryptionkey@0"
I20260926 23:28:49.052974 21669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:49.059532 21669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:49.064918 21669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:49.065285 21669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:49.085006 21669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37483
I20260926 23:28:49.085048 21735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37483 every 8 connection(s)
I20260926 23:28:49.086608 21669 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:49.091013 21736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:49.093037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21669
I20260926 23:28:49.093218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:49.093700 388 external_mini_cluster.cc:1428] Setting key 9037f0bdc6fc7aeb9b23ac685b797d4d
I20260926 23:28:49.104328 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:49.095933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:39737 (local address 127.0.0.1:37483)
0926 23:28:49.097079 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:49.097096 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:49.097532 (+ 436us) server_negotiation.cc:407] Connection header received
0926 23:28:49.099982 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:49.100015 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:49.100517 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:49.101060 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:49.102170 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:49.102179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:49.102197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:49.102213 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:49.102292 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:49.102320 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:49.102579 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:49.102739 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 23:28:49.102974 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":489,"threads_started":1}
I20260926 23:28:49.105307 21736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032: Bootstrap starting.
I20260926 23:28:49.108639 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:49.109058 21736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:49.110508 21736 log.cc:824] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:49.114751 21736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032: No bootstrap required, opened a new log
I20260926 23:28:49.117841 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:49.114719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:34649 (local address 127.0.0.1:37483)
0926 23:28:49.114924 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:49.114931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:49.115511 (+ 580us) server_negotiation.cc:407] Connection header received
0926 23:28:49.115736 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:49.115740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:49.115801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:49.115895 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:49.117066 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:49.117070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:49.117074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:49.117095 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:49.117148 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:49.117165 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:49.117285 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:49.117422 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:28:49.117486 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:28:49.125291 21736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5a84248cd647b9865b6da57aadc032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260926 23:28:49.125752 21736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:49.125886 21736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5a84248cd647b9865b6da57aadc032, State: Initialized, Role: FOLLOWER
I20260926 23:28:49.126557 21736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5a84248cd647b9865b6da57aadc032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260926 23:28:49.126829 21736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:49.126999 21736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:49.127190 21736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:49.129911 21736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5a84248cd647b9865b6da57aadc032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260926 23:28:49.130553 21736 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5a84248cd647b9865b6da57aadc032; no voters:
I20260926 23:28:49.131251 21736 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:49.131431 21746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:49.132241 21746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [term 1 LEADER]: Becoming Leader. State: Replica: 9a5a84248cd647b9865b6da57aadc032, State: Running, Role: LEADER
I20260926 23:28:49.132941 21746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5a84248cd647b9865b6da57aadc032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260926 23:28:49.133983 21736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:49.136535 21747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5a84248cd647b9865b6da57aadc032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5a84248cd647b9865b6da57aadc032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260926 23:28:49.136548 21748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5a84248cd647b9865b6da57aadc032. Latest consensus state: current_term: 1 leader_uuid: "9a5a84248cd647b9865b6da57aadc032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5a84248cd647b9865b6da57aadc032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260926 23:28:49.137136 21747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:49.137136 21748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:49.138125 21752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:49.146255 21752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:49.157881 21752 catalog_manager.cc:1347] Generated new cluster ID: 114892e7656d4ea195089d13953f128a
I20260926 23:28:49.158036 21752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:49.176551 21752 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:49.177134 21752 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:49.187750 21752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5a84248cd647b9865b6da57aadc032: Generated new TSK 0
I20260926 23:28:49.188910 21752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:49.208227 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21669
W20260926 23:28:49.209555 21731 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:39737
W20260926 23:28:49.210487 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21669: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:49.210680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21669
2026-09-26T23:28:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:49 dist-test-slave-q5g8 krb5kdc[21767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:49 dist-test-slave-q5g8 krb5kdc[21767](info): set up 2 sockets
Sep 26 23:28:49 dist-test-slave-q5g8 krb5kdc[21767](info): commencing operation
krb5kdc: starting...
W20260926 23:28:50.385244 388 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:50 dist-test-slave-q5g8 krb5kdc[21767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:50.685039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:51.047053 21780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:51.047395 21780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:51.047570 21780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:51.055970 21780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:51.056118 21780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:51.056214 21780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:51.056316 21780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:51.056424 21780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:51.067224 21780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:51.068830 21780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:51.070403 21780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:51.085059 21785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:51.085120 21786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:51.085088 21788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:51.086110 21780 server_base.cc:1034] running on GCE node
I20260926 23:28:51.087038 21780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:51.088900 21780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:51.090197 21780 hybrid_clock.cc:648] HybridClock initialized: now 1790465331090108 us; error 73 us; skew 500 ppm
I20260926 23:28:51.090744 21780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:51 dist-test-slave-q5g8 krb5kdc[21767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:51.099435 21780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:51.101527 21780 webserver.cc:460] Webserver started at http://127.0.0.1:41673/ using document root <none> and password file <none>
I20260926 23:28:51.102523 21780 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:51.102653 21780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:51.102957 21780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:51.106568 21780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "9f39612b133c4ef2b21105a3ba8c843a"
format_stamp: "Formatted at 2026-09-26 23:28:51 on dist-test-slave-q5g8"
server_key: "49dbf9ec14ae428aefee5f50fb2f20dd"
server_key_iv: "641e689e0c6e478ccafbbc64a0a3b4c1"
server_key_version: "encryptionkey@0"
I20260926 23:28:51.107487 21780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "9f39612b133c4ef2b21105a3ba8c843a"
format_stamp: "Formatted at 2026-09-26 23:28:51 on dist-test-slave-q5g8"
server_key: "49dbf9ec14ae428aefee5f50fb2f20dd"
server_key_iv: "641e689e0c6e478ccafbbc64a0a3b4c1"
server_key_version: "encryptionkey@0"
I20260926 23:28:51.114023 21780 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 23:28:51.118523 21795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:51.120442 21780 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:28:51.120657 21780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "9f39612b133c4ef2b21105a3ba8c843a"
format_stamp: "Formatted at 2026-09-26 23:28:51 on dist-test-slave-q5g8"
server_key: "49dbf9ec14ae428aefee5f50fb2f20dd"
server_key_iv: "641e689e0c6e478ccafbbc64a0a3b4c1"
server_key_version: "encryptionkey@0"
I20260926 23:28:51.121045 21780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:51.141496 21780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:51.146944 21780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:51.147287 21780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:51.169050 21780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42745
I20260926 23:28:51.169091 21846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42745 every 8 connection(s)
I20260926 23:28:51.170714 21780 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:51.173938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21780
I20260926 23:28:51.174165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:51.174649 388 external_mini_cluster.cc:1428] Setting key 63f1d3c63e8468a0c5c4757ad1050af7
I20260926 23:28:51.175402 21847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:51.189939 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a: Bootstrap starting.
Sep 26 23:28:51 dist-test-slave-q5g8 krb5kdc[21767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:28:51.193831 21847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:51.195075 21847 log.cc:824] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:51.198410 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a: No bootstrap required, opened a new log
I20260926 23:28:51.200299 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:51.176664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:60503 (local address 127.0.0.1:42745)
0926 23:28:51.177746 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:51.177765 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:51.178151 (+ 386us) server_negotiation.cc:407] Connection header received
0926 23:28:51.180068 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:51.180096 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:51.180626 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:51.181191 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:51.182860 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:51.184279 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:51.185882 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:51.186775 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:51.187071 (+ 296us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:51.191788 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:51.191797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:51.191814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:51.191873 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:51.195472 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:51.196463 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:51.196471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:51.196479 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:51.196665 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:51.197236 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:51.197248 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:51.197250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:51.197814 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:51.198218 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:51.198784 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 23:28:51.199154 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":441,"threads_started":1}
I20260926 23:28:51.204836 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:51.207340 21847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f39612b133c4ef2b21105a3ba8c843a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 23:28:51.207841 21847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:51.207966 21847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f39612b133c4ef2b21105a3ba8c843a, State: Initialized, Role: FOLLOWER
I20260926 23:28:51.208667 21847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f39612b133c4ef2b21105a3ba8c843a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 23:28:51.209012 21847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:51.209168 21847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:51.209412 21847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:51.213140 21847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f39612b133c4ef2b21105a3ba8c843a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 23:28:51.213809 21847 leader_election.cc:302] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f39612b133c4ef2b21105a3ba8c843a; no voters:
I20260926 23:28:51.217168 21847 leader_election.cc:288] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:51.217308 21857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:51.218968 21857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [term 1 LEADER]: Becoming Leader. State: Replica: 9f39612b133c4ef2b21105a3ba8c843a, State: Running, Role: LEADER
I20260926 23:28:51.219658 21857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f39612b133c4ef2b21105a3ba8c843a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 23:28:51.220006 21847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:51.224773 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f39612b133c4ef2b21105a3ba8c843a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f39612b133c4ef2b21105a3ba8c843a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } } }
I20260926 23:28:51.225419 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:51.228950 21859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f39612b133c4ef2b21105a3ba8c843a. Latest consensus state: current_term: 1 leader_uuid: "9f39612b133c4ef2b21105a3ba8c843a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f39612b133c4ef2b21105a3ba8c843a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } } }
I20260926 23:28:51.229887 21859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:51.233256 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:51.209945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:38771 (local address 127.0.0.1:42745)
0926 23:28:51.210218 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:51.210229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:51.210756 (+ 527us) server_negotiation.cc:407] Connection header received
0926 23:28:51.210954 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:51.210958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:51.211036 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:51.211150 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:51.213156 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:51.214174 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:51.216826 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:51.217413 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:51.217576 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:51.221097 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:51.221102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:51.221106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:51.221151 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:51.226995 (+ 5844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:51.228642 (+ 1647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:51.228650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:51.228658 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:51.228939 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:51.229829 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:51.229837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:51.229843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:51.230412 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:51.232178 (+ 1766us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:51.232447 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 23:28:51.232753 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:28:51.237859 21865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:51.243893 21865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:51.254911 21865 catalog_manager.cc:1347] Generated new cluster ID: 8535c15883f844e6ab3a71b360266175
I20260926 23:28:51.255048 21865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:51.274641 21865 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:51.276870 21865 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:51.289862 21865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f39612b133c4ef2b21105a3ba8c843a: Generated new TSK 0
I20260926 23:28:51.291159 21865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:51.309371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21780
I20260926 23:28:51.310966 21842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:51.409879 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21780
W20260926 23:28:51.428010 21765 connection.cc:535] client connection to 127.0.0.1:42745 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42745): Connection reset by peer (error 104)
2026-09-26T23:28:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:51 dist-test-slave-q5g8 krb5kdc[21879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:51 dist-test-slave-q5g8 krb5kdc[21879](info): set up 2 sockets
Sep 26 23:28:51 dist-test-slave-q5g8 krb5kdc[21879](info): commencing operation
krb5kdc: starting...
W20260926 23:28:52.589640 388 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:52 dist-test-slave-q5g8 krb5kdc[21879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:52.880617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:53.242771 21892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:53.243216 21892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:53.243412 21892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:53.251426 21892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:53.251626 21892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:53.251746 21892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:53.251868 21892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:53.251958 21892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:53.262832 21892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:53.264451 21892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:53.266017 21892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:53.280019 21898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:53.280064 21900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:53.280092 21897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:53.281001 21892 server_base.cc:1034] running on GCE node
I20260926 23:28:53.282033 21892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:53.283782 21892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:53.285015 21892 hybrid_clock.cc:648] HybridClock initialized: now 1790465333284946 us; error 67 us; skew 500 ppm
I20260926 23:28:53.285439 21892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:53 dist-test-slave-q5g8 krb5kdc[21879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:53.289368 21892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:53.291137 21892 webserver.cc:460] Webserver started at http://127.0.0.1:33499/ using document root <none> and password file <none>
I20260926 23:28:53.291927 21892 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:53.292042 21892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:53.292339 21892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:53.296525 21892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "eacbbc6bf97343fd92b2e06357659b98"
format_stamp: "Formatted at 2026-09-26 23:28:53 on dist-test-slave-q5g8"
server_key: "fd74f3f59c554951244236c848237c13"
server_key_iv: "5c106c74778554066a9bb2dc42410c3c"
server_key_version: "encryptionkey@0"
I20260926 23:28:53.297605 21892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "eacbbc6bf97343fd92b2e06357659b98"
format_stamp: "Formatted at 2026-09-26 23:28:53 on dist-test-slave-q5g8"
server_key: "fd74f3f59c554951244236c848237c13"
server_key_iv: "5c106c74778554066a9bb2dc42410c3c"
server_key_version: "encryptionkey@0"
I20260926 23:28:53.303670 21892 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:28:53.307783 21907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:53.309532 21892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:28:53.309783 21892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "eacbbc6bf97343fd92b2e06357659b98"
format_stamp: "Formatted at 2026-09-26 23:28:53 on dist-test-slave-q5g8"
server_key: "fd74f3f59c554951244236c848237c13"
server_key_iv: "5c106c74778554066a9bb2dc42410c3c"
server_key_version: "encryptionkey@0"
I20260926 23:28:53.310021 21892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:53.334300 21892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:53.339277 21892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:53.339593 21892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:53.358983 21892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36715
I20260926 23:28:53.359143 21958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36715 every 8 connection(s)
I20260926 23:28:53.360419 21892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:53.364627 21959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:53.369490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 21892
I20260926 23:28:53.369748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:53.370199 388 external_mini_cluster.cc:1428] Setting key d75ed9dfb67f637b0e681ce262095639
I20260926 23:28:53.377275 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98: Bootstrap starting.
I20260926 23:28:53.381553 21959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:53.383116 21959 log.cc:824] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:53.385771 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:53.372243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:47213 (local address 127.0.0.1:36715)
0926 23:28:53.373095 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:53.373108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:53.373155 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:28:53.375310 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:53.375338 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:53.375854 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:53.376238 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:53.377040 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:53.378467 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:53.380554 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:53.381396 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:53.381553 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:53.383002 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:53.383010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:53.383025 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:53.383800 (+ 775us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:53.383918 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:53.383957 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:53.384143 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:53.384471 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 23:28:53.384755 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":406,"threads_started":1}
I20260926 23:28:53.386673 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98: No bootstrap required, opened a new log
I20260926 23:28:53.390060 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:53.396036 21959 raft_consensus.cc:348] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacbbc6bf97343fd92b2e06357659b98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36715 } }
I20260926 23:28:53.396557 21959 raft_consensus.cc:374] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:53.396714 21959 raft_consensus.cc:729] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacbbc6bf97343fd92b2e06357659b98, State: Initialized, Role: FOLLOWER
I20260926 23:28:53.397468 21959 consensus_queue.cc:260] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacbbc6bf97343fd92b2e06357659b98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36715 } }
I20260926 23:28:53.397758 21959 raft_consensus.cc:388] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:53.397930 21959 raft_consensus.cc:482] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:53.398135 21959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:53.400782 21959 raft_consensus.cc:504] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacbbc6bf97343fd92b2e06357659b98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36715 } }
I20260926 23:28:53.401520 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:53.394594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:45573 (local address 127.0.0.1:36715)
0926 23:28:53.394796 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:53.394800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:53.395290 (+ 490us) server_negotiation.cc:407] Connection header received
0926 23:28:53.395521 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:53.395525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:53.395596 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:53.395706 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:53.396281 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:53.397280 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:53.398765 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:53.399294 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:53.399461 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:53.400652 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:53.400657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:53.400660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:53.400723 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:53.400775 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:53.400802 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:53.400928 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:53.401156 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 23:28:53.401211 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:28:53.401502 21959 leader_election.cc:302] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eacbbc6bf97343fd92b2e06357659b98; no voters:
I20260926 23:28:53.402383 21959 leader_election.cc:288] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:53.402513 21969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:53.403611 21969 raft_consensus.cc:686] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [term 1 LEADER]: Becoming Leader. State: Replica: eacbbc6bf97343fd92b2e06357659b98, State: Running, Role: LEADER
I20260926 23:28:53.404266 21969 consensus_queue.cc:237] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacbbc6bf97343fd92b2e06357659b98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36715 } }
I20260926 23:28:53.405050 21959 sys_catalog.cc:564] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:53.407342 21971 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eacbbc6bf97343fd92b2e06357659b98. Latest consensus state: current_term: 1 leader_uuid: "eacbbc6bf97343fd92b2e06357659b98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacbbc6bf97343fd92b2e06357659b98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36715 } } }
I20260926 23:28:53.407773 21971 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:53.407701 21970 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eacbbc6bf97343fd92b2e06357659b98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacbbc6bf97343fd92b2e06357659b98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36715 } } }
I20260926 23:28:53.409428 21970 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:53.413486 21974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:53.420481 21974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:53.431325 21974 catalog_manager.cc:1347] Generated new cluster ID: 3786fac39a2a4e638c6998bfc9a3ab99
I20260926 23:28:53.431460 21974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:53.462698 21974 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:53.464126 21974 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:53.479422 21974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eacbbc6bf97343fd92b2e06357659b98: Generated new TSK 0
I20260926 23:28:53.480779 21974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:53.490911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21892
W20260926 23:28:53.492134 21954 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:47213
W20260926 23:28:53.493021 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 21892: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:53.493196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 21892
W20260926 23:28:53.513723 21877 connection.cc:535] client connection to 127.0.0.1:36715 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36715): Connection reset by peer (error 104)
2026-09-26T23:28:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2081 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:53 dist-test-slave-q5g8 krb5kdc[21990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:53 dist-test-slave-q5g8 krb5kdc[21990](info): set up 2 sockets
Sep 26 23:28:53 dist-test-slave-q5g8 krb5kdc[21990](info): commencing operation
krb5kdc: starting...
W20260926 23:28:54.665570 388 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:54 dist-test-slave-q5g8 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:28:54.956971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:55.331573 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:55.331880 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:55.332049 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:55.339636 22003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:55.339769 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:55.339860 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:55.339946 22003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:55.340009 22003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:55.350981 22003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:55.352481 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:55.353992 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:55.367518 22008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:55.367558 22011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:55.367494 22009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:55.368515 22003 server_base.cc:1034] running on GCE node
I20260926 23:28:55.369567 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:55.371340 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:55.372582 22003 hybrid_clock.cc:648] HybridClock initialized: now 1790465335372535 us; error 38 us; skew 500 ppm
I20260926 23:28:55.373072 22003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:55 dist-test-slave-q5g8 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:55.376513 22003 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:28:55.378320 22003 webserver.cc:460] Webserver started at http://127.0.0.1:34493/ using document root <none> and password file <none>
I20260926 23:28:55.379083 22003 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:55.379251 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:55.379576 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:55.382817 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "c34f2554695d4e68ba2ff00cea4e02d6"
format_stamp: "Formatted at 2026-09-26 23:28:55 on dist-test-slave-q5g8"
server_key: "066a7cd8952ee19e70ca463515b36210"
server_key_iv: "41a80836e4ba48c6fd73d9b8a78bd98a"
server_key_version: "encryptionkey@0"
I20260926 23:28:55.383641 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "c34f2554695d4e68ba2ff00cea4e02d6"
format_stamp: "Formatted at 2026-09-26 23:28:55 on dist-test-slave-q5g8"
server_key: "066a7cd8952ee19e70ca463515b36210"
server_key_iv: "41a80836e4ba48c6fd73d9b8a78bd98a"
server_key_version: "encryptionkey@0"
I20260926 23:28:55.389501 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:28:55.393911 22018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:55.395780 22003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:28:55.396026 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "c34f2554695d4e68ba2ff00cea4e02d6"
format_stamp: "Formatted at 2026-09-26 23:28:55 on dist-test-slave-q5g8"
server_key: "066a7cd8952ee19e70ca463515b36210"
server_key_iv: "41a80836e4ba48c6fd73d9b8a78bd98a"
server_key_version: "encryptionkey@0"
I20260926 23:28:55.396310 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:55.403411 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:55.408236 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:55.408583 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:55.429855 22003 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33361
I20260926 23:28:55.429911 22069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33361 every 8 connection(s)
I20260926 23:28:55.431514 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:55.435933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22003
I20260926 23:28:55.436033 22070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:55.436154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:55.436738 388 external_mini_cluster.cc:1428] Setting key 2c4056f2bf04cbb45ae06c1f3f99483a
I20260926 23:28:55.446017 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:55.438724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:37405 (local address 127.0.0.1:33361)
0926 23:28:55.439650 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:55.439664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:55.439715 (+ 51us) server_negotiation.cc:407] Connection header received
0926 23:28:55.441817 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:55.441846 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:55.442306 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:55.442804 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:55.444100 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:55.444107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:55.444123 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:55.444141 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:55.444220 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:55.444246 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:55.444536 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:55.444729 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 23:28:55.444959 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":395,"threads_started":1}
I20260926 23:28:55.450907 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:55.452157 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6: Bootstrap starting.
I20260926 23:28:55.456261 22070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:55.457990 22070 log.cc:824] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:55.459527 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:55.455985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:56909 (local address 127.0.0.1:33361)
0926 23:28:55.456271 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:55.456280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:55.456815 (+ 535us) server_negotiation.cc:407] Connection header received
0926 23:28:55.457046 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:55.457050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:55.457109 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:55.457210 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:55.458564 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:55.458569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:55.458572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:55.458685 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:55.458760 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:55.458790 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:55.458971 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:55.459078 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 23:28:55.459168 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 23:28:55.462131 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6: No bootstrap required, opened a new log
I20260926 23:28:55.470477 22070 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33361 } }
I20260926 23:28:55.470935 22070 raft_consensus.cc:374] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:55.471107 22070 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34f2554695d4e68ba2ff00cea4e02d6, State: Initialized, Role: FOLLOWER
I20260926 23:28:55.471805 22070 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33361 } }
I20260926 23:28:55.472049 22070 raft_consensus.cc:388] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:55.472276 22070 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:55.472462 22070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:55.474974 22070 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33361 } }
I20260926 23:28:55.475627 22070 leader_election.cc:302] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c34f2554695d4e68ba2ff00cea4e02d6; no voters:
I20260926 23:28:55.476740 22070 leader_election.cc:288] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:55.476866 22080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:55.477761 22080 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [term 1 LEADER]: Becoming Leader. State: Replica: c34f2554695d4e68ba2ff00cea4e02d6, State: Running, Role: LEADER
I20260926 23:28:55.478495 22080 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33361 } }
I20260926 23:28:55.479378 22070 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:55.482286 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34f2554695d4e68ba2ff00cea4e02d6. Latest consensus state: current_term: 1 leader_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33361 } } }
I20260926 23:28:55.482749 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:55.482178 22081 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34f2554695d4e68ba2ff00cea4e02d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33361 } } }
I20260926 23:28:55.485086 22081 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:55.487857 22089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:55.492995 22089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:55.503640 22089 catalog_manager.cc:1347] Generated new cluster ID: 24e1f97415314e9eb79f03c59f6f9376
I20260926 23:28:55.503789 22089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:55.535163 22089 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:55.535758 22089 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:55.543344 22089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c34f2554695d4e68ba2ff00cea4e02d6: Generated new TSK 0
I20260926 23:28:55.544659 22089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:55.554618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22003
W20260926 23:28:55.556622 22065 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:37405
W20260926 23:28:55.557504 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22003: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:55.557684 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22003
2026-09-26T23:28:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2058 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:55 dist-test-slave-q5g8 krb5kdc[22101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:55 dist-test-slave-q5g8 krb5kdc[22101](info): set up 2 sockets
Sep 26 23:28:55 dist-test-slave-q5g8 krb5kdc[22101](info): commencing operation
krb5kdc: starting...
W20260926 23:28:56.721081 388 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:56 dist-test-slave-q5g8 krb5kdc[22101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:28:57.005002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34067
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:57.357270 22114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:57.357666 22114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:57.357832 22114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:57.365842 22114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:57.366029 22114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:57.366145 22114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:57.366251 22114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:57.366339 22114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:57.377864 22114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:57.379554 22114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:57.381381 22114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:57.394842 22119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:57.394914 22122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:57.394910 22120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:57.395859 22114 server_base.cc:1034] running on GCE node
I20260926 23:28:57.396857 22114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:57.398530 22114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:57.399837 22114 hybrid_clock.cc:648] HybridClock initialized: now 1790465337399785 us; error 65 us; skew 500 ppm
I20260926 23:28:57.400256 22114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:57 dist-test-slave-q5g8 krb5kdc[22101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:57.410458 22114 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:28:57.412297 22114 webserver.cc:460] Webserver started at http://127.0.97.62:37993/ using document root <none> and password file <none>
I20260926 23:28:57.413172 22114 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:57.413336 22114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:57.413656 22114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:57.416962 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "73370f9b8fe4444c9a3e5535336c508b"
format_stamp: "Formatted at 2026-09-26 23:28:57 on dist-test-slave-q5g8"
server_key: "2b06bca1dae942ae3e51d216ee81842d"
server_key_iv: "19ff6583a685263ca5b315968be7fa50"
server_key_version: "encryptionkey@0"
I20260926 23:28:57.417824 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "73370f9b8fe4444c9a3e5535336c508b"
format_stamp: "Formatted at 2026-09-26 23:28:57 on dist-test-slave-q5g8"
server_key: "2b06bca1dae942ae3e51d216ee81842d"
server_key_iv: "19ff6583a685263ca5b315968be7fa50"
server_key_version: "encryptionkey@0"
I20260926 23:28:57.423626 22114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 23:28:57.427470 22129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:57.428958 22114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:28:57.429163 22114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "73370f9b8fe4444c9a3e5535336c508b"
format_stamp: "Formatted at 2026-09-26 23:28:57 on dist-test-slave-q5g8"
server_key: "2b06bca1dae942ae3e51d216ee81842d"
server_key_iv: "19ff6583a685263ca5b315968be7fa50"
server_key_version: "encryptionkey@0"
I20260926 23:28:57.429375 22114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:57.464613 22114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:57.469797 22114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:57.470089 22114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:57.489131 22114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34067
I20260926 23:28:57.489151 22180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34067 every 8 connection(s)
I20260926 23:28:57.490701 22114 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:57.492234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22114
I20260926 23:28:57.492410 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:57.492906 388 external_mini_cluster.cc:1428] Setting key 012c968bf0c36884147bf83cc4abae07
I20260926 23:28:57.495308 22181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:28:57 dist-test-slave-q5g8 krb5kdc[22101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:28:57.509642 22181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b: Bootstrap starting.
I20260926 23:28:57.512960 22181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:57.514056 22181 log.cc:824] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:57.516916 22181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b: No bootstrap required, opened a new log
I20260926 23:28:57.518798 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:57.494811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60207 (local address 127.0.97.62:34067)
0926 23:28:57.495831 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:57.495848 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:57.496155 (+ 307us) server_negotiation.cc:407] Connection header received
0926 23:28:57.498090 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:57.498116 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:57.498564 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:57.498935 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:57.500774 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:57.502366 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:57.504167 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:57.505101 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:57.505258 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:57.510533 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:57.510542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:57.510558 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:57.510638 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:57.514174 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:57.515141 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:57.515149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:57.515160 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:57.515331 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:57.515841 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:57.515852 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:57.515854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:57.516396 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:57.516784 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:57.517306 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 23:28:57.517665 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":417,"threads_started":1}
I20260926 23:28:57.523335 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:57.525847 22181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73370f9b8fe4444c9a3e5535336c508b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34067 } }
I20260926 23:28:57.526257 22181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:57.526373 22181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73370f9b8fe4444c9a3e5535336c508b, State: Initialized, Role: FOLLOWER
I20260926 23:28:57.527130 22181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73370f9b8fe4444c9a3e5535336c508b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34067 } }
I20260926 23:28:57.527480 22181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:57.527653 22181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:57.527913 22181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:57.531653 22181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73370f9b8fe4444c9a3e5535336c508b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34067 } }
I20260926 23:28:57.532343 22181 leader_election.cc:302] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73370f9b8fe4444c9a3e5535336c508b; no voters:
I20260926 23:28:57.533034 22181 leader_election.cc:288] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:57.533166 22191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:57.535346 22181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:57.540149 22191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [term 1 LEADER]: Becoming Leader. State: Replica: 73370f9b8fe4444c9a3e5535336c508b, State: Running, Role: LEADER
I20260926 23:28:57.540920 22191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73370f9b8fe4444c9a3e5535336c508b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34067 } }
I20260926 23:28:57.544065 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73370f9b8fe4444c9a3e5535336c508b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73370f9b8fe4444c9a3e5535336c508b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34067 } } }
I20260926 23:28:57.544499 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:57.548806 22203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:57.548583 22193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73370f9b8fe4444c9a3e5535336c508b. Latest consensus state: current_term: 1 leader_uuid: "73370f9b8fe4444c9a3e5535336c508b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73370f9b8fe4444c9a3e5535336c508b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34067 } } }
I20260926 23:28:57.549090 22193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:57.550172 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:57.528409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33537 (local address 127.0.97.62:34067)
0926 23:28:57.528746 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:57.528752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:57.529515 (+ 763us) server_negotiation.cc:407] Connection header received
0926 23:28:57.529798 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:57.529847 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:57.529969 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:57.530077 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:57.532115 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:57.533357 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:57.535845 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:57.536522 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:57.536737 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:57.541043 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:57.541053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:57.541058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:28:57.541104 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:57.545551 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:57.546869 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:57.546877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:57.546883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:57.547131 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:28:57.547835 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:28:57.547844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:28:57.547846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:28:57.548250 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:57.548600 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:57.549748 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0926 23:28:57.549893 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260926 23:28:57.556542 22203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:57.566890 22203 catalog_manager.cc:1347] Generated new cluster ID: 9dbb5a4ce7924966b0435babe7324e20
I20260926 23:28:57.567027 22203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:57.590137 22203 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:57.591645 22203 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:57.611877 22203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73370f9b8fe4444c9a3e5535336c508b: Generated new TSK 0
I20260926 23:28:57.613128 22203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:57.631675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22114
I20260926 23:28:57.632917 22176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:28:57.728417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22114
W20260926 23:28:57.746083 22099 connection.cc:535] client connection to 127.0.97.62:34067 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:34067): Connection reset by peer (error 104)
2026-09-26T23:28:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2172 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:57 dist-test-slave-q5g8 krb5kdc[22214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:57 dist-test-slave-q5g8 krb5kdc[22214](info): set up 2 sockets
Sep 26 23:28:57 dist-test-slave-q5g8 krb5kdc[22214](info): commencing operation
krb5kdc: starting...
W20260926 23:28:58.845152 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:28:59 dist-test-slave-q5g8 krb5kdc[22214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:28:59.112371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43959
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:28:59.445480 22227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:28:59.445778 22227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:28:59.445963 22227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:28:59.453142 22227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:28:59.453294 22227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:28:59.453384 22227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:28:59.453483 22227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:28:59.453572 22227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:28:59.463542 22227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:28:59.464933 22227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:28:59.466264 22227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:28:59.477903 22232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:59.477901 22235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:28:59.478170 22233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:28:59.479615 22227 server_base.cc:1034] running on GCE node
I20260926 23:28:59.480396 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:28:59.481930 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:28:59.483165 22227 hybrid_clock.cc:648] HybridClock initialized: now 1790465339483128 us; error 62 us; skew 500 ppm
I20260926 23:28:59.483513 22227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:28:59 dist-test-slave-q5g8 krb5kdc[22214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:28:59.486639 22227 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:28:59.488154 22227 webserver.cc:460] Webserver started at http://127.0.97.62:45953/ using document root <none> and password file <none>
I20260926 23:28:59.488896 22227 fs_manager.cc:359] Metadata directory not provided
I20260926 23:28:59.489032 22227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:28:59.489332 22227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:28:59.492271 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "e394c9e2ceb94f8798c7f2aa97968d19"
format_stamp: "Formatted at 2026-09-26 23:28:59 on dist-test-slave-q5g8"
server_key: "f47d2e5898bc03ff61e93e19d08430af"
server_key_iv: "4c9896b3d2e4afa5aa6a161b6f6f68be"
server_key_version: "encryptionkey@0"
I20260926 23:28:59.493088 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "e394c9e2ceb94f8798c7f2aa97968d19"
format_stamp: "Formatted at 2026-09-26 23:28:59 on dist-test-slave-q5g8"
server_key: "f47d2e5898bc03ff61e93e19d08430af"
server_key_iv: "4c9896b3d2e4afa5aa6a161b6f6f68be"
server_key_version: "encryptionkey@0"
I20260926 23:28:59.498376 22227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:28:59.501875 22242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:28:59.503166 22227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:28:59.503341 22227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "e394c9e2ceb94f8798c7f2aa97968d19"
format_stamp: "Formatted at 2026-09-26 23:28:59 on dist-test-slave-q5g8"
server_key: "f47d2e5898bc03ff61e93e19d08430af"
server_key_iv: "4c9896b3d2e4afa5aa6a161b6f6f68be"
server_key_version: "encryptionkey@0"
I20260926 23:28:59.503528 22227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:28:59.526216 22227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:28:59.530714 22227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:28:59.531010 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:28:59.551147 22227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43959
I20260926 23:28:59.551179 22293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43959 every 8 connection(s)
I20260926 23:28:59.552506 22227 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:28:59.556138 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:28:59.557008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22227
I20260926 23:28:59.557194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:28:59.557602 388 external_mini_cluster.cc:1428] Setting key de570472b29629d54bc31433faae1a85
I20260926 23:28:59.569710 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19: Bootstrap starting.
I20260926 23:28:59.571316 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:59.559511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45017 (local address 127.0.97.62:43959)
0926 23:28:59.560308 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:59.560322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:59.560365 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:28:59.562071 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:59.562094 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:59.562493 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:59.562823 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:59.563390 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:59.564754 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:59.566693 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:59.567434 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:59.567570 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:59.568934 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:59.568942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:59.568955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:59.569650 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:59.569734 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:59.569760 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:59.569920 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:59.570194 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 23:28:59.570363 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":362,"threads_started":1}
I20260926 23:28:59.573058 22294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19: Neither blocks nor log segments found. Creating new log.
I20260926 23:28:59.574251 22294 log.cc:824] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19: Log is configured to *not* fsync() on all Append() calls
I20260926 23:28:59.575063 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:28:59.577536 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19: No bootstrap required, opened a new log
I20260926 23:28:59.587330 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:28:59.580261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39813 (local address 127.0.97.62:43959)
0926 23:28:59.580483 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:28:59.580497 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:28:59.580996 (+ 499us) server_negotiation.cc:407] Connection header received
0926 23:28:59.581241 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:28:59.581246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:28:59.581327 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:28:59.581423 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:28:59.581983 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:59.582784 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:59.584775 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:28:59.585269 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:28:59.585419 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:28:59.586609 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:28:59.586614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:28:59.586617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:28:59.586645 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:28:59.586684 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:28:59.586699 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:28:59.586816 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:28:59.587003 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 23:28:59.587059 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:28:59.587642 22294 raft_consensus.cc:348] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 23:28:59.588117 22294 raft_consensus.cc:374] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:28:59.588250 22294 raft_consensus.cc:729] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e394c9e2ceb94f8798c7f2aa97968d19, State: Initialized, Role: FOLLOWER
I20260926 23:28:59.589012 22294 consensus_queue.cc:260] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 23:28:59.589264 22294 raft_consensus.cc:388] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:28:59.589368 22294 raft_consensus.cc:482] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:28:59.589495 22294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:28:59.592114 22294 raft_consensus.cc:504] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 23:28:59.592587 22294 leader_election.cc:302] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e394c9e2ceb94f8798c7f2aa97968d19; no voters:
I20260926 23:28:59.593303 22294 leader_election.cc:288] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:28:59.593397 22304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:28:59.594234 22304 raft_consensus.cc:686] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [term 1 LEADER]: Becoming Leader. State: Replica: e394c9e2ceb94f8798c7f2aa97968d19, State: Running, Role: LEADER
I20260926 23:28:59.594893 22304 consensus_queue.cc:237] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 23:28:59.595757 22294 sys_catalog.cc:564] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:28:59.597975 22306 sys_catalog.cc:455] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e394c9e2ceb94f8798c7f2aa97968d19. Latest consensus state: current_term: 1 leader_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } } }
I20260926 23:28:59.598439 22306 sys_catalog.cc:458] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:59.601202 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:28:59.601182 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e394c9e2ceb94f8798c7f2aa97968d19" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } } }
I20260926 23:28:59.601969 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19 [sys.catalog]: This master's current role is: LEADER
I20260926 23:28:59.608294 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:28:59.617347 22312 catalog_manager.cc:1347] Generated new cluster ID: 15b5ba89324643acbe8ff58be7c4d138
I20260926 23:28:59.617475 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:28:59.652372 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:28:59.654037 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:28:59.659394 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e394c9e2ceb94f8798c7f2aa97968d19: Generated new TSK 0
I20260926 23:28:59.660174 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:28:59.670554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22227
W20260926 23:28:59.671707 22289 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45017
W20260926 23:28:59.672649 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22227: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:28:59.672860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22227
W20260926 23:28:59.689877 22212 connection.cc:535] client connection to 127.0.97.62:43959 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:43959): Connection reset by peer (error 104)
2026-09-26T23:28:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:28:59 dist-test-slave-q5g8 krb5kdc[22325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:28:59 dist-test-slave-q5g8 krb5kdc[22325](info): set up 2 sockets
Sep 26 23:28:59 dist-test-slave-q5g8 krb5kdc[22325](info): commencing operation
krb5kdc: starting...
W20260926 23:29:00.793226 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.009s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:29:00 dist-test-slave-q5g8 krb5kdc[22325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:29:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:29:01.069067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39959
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:29:01.420591 22338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:01.420948 22338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:01.421130 22338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:01.428720 22338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:01.428846 22338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:01.428929 22338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:01.429044 22338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:01.429111 22338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:01.440385 22338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:01.441882 22338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:01.443277 22338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:01.455873 22344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:01.456156 22346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:01.456418 22343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:01.456959 22338 server_base.cc:1034] running on GCE node
I20260926 23:29:01.457814 22338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:01.459352 22338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:01.460736 22338 hybrid_clock.cc:648] HybridClock initialized: now 1790465341460620 us; error 43 us; skew 500 ppm
I20260926 23:29:01.461186 22338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:01 dist-test-slave-q5g8 krb5kdc[22325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:01.464998 22338 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:29:01.466881 22338 webserver.cc:460] Webserver started at http://127.0.97.62:37007/ using document root <none> and password file <none>
I20260926 23:29:01.467733 22338 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:01.467898 22338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:01.468245 22338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:01.471854 22338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "a7e1751889ec419195181ea1812b9cb5"
format_stamp: "Formatted at 2026-09-26 23:29:01 on dist-test-slave-q5g8"
server_key: "56720fe9079a74c485ffd29e72695207"
server_key_iv: "1eb687bb1fe7a9b6892bf876f4b08c34"
server_key_version: "encryptionkey@0"
I20260926 23:29:01.472862 22338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "a7e1751889ec419195181ea1812b9cb5"
format_stamp: "Formatted at 2026-09-26 23:29:01 on dist-test-slave-q5g8"
server_key: "56720fe9079a74c485ffd29e72695207"
server_key_iv: "1eb687bb1fe7a9b6892bf876f4b08c34"
server_key_version: "encryptionkey@0"
I20260926 23:29:01.479218 22338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:29:01.483093 22353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:01.484565 22338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:29:01.484849 22338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "a7e1751889ec419195181ea1812b9cb5"
format_stamp: "Formatted at 2026-09-26 23:29:01 on dist-test-slave-q5g8"
server_key: "56720fe9079a74c485ffd29e72695207"
server_key_iv: "1eb687bb1fe7a9b6892bf876f4b08c34"
server_key_version: "encryptionkey@0"
I20260926 23:29:01.485073 22338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:01.491209 22338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:01.496002 22338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:01.496327 22338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:01.515910 22338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39959
I20260926 23:29:01.515964 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39959 every 8 connection(s)
I20260926 23:29:01.517537 22338 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:29:01.521338 22405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:01.526662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22338
I20260926 23:29:01.526834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:29:01.527266 388 external_mini_cluster.cc:1428] Setting key 7c5825c32db05eeeafd5f8b45843782d
I20260926 23:29:01.534133 22405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5: Bootstrap starting.
I20260926 23:29:01.536939 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:01.529264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52719 (local address 127.0.97.62:39959)
0926 23:29:01.530128 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:01.530144 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:01.530197 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:29:01.532745 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:01.532778 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:01.533255 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:01.533742 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:01.534774 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:01.534787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:01.534811 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:29:01.534833 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:01.534903 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:29:01.534932 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:01.535157 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:01.535294 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:29:01.535491 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":363,"threads_started":1}
I20260926 23:29:01.538288 22405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:01.539433 22405 log.cc:824] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:01.541196 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:29:01.542618 22405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5: No bootstrap required, opened a new log
I20260926 23:29:01.550141 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:01.546036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47403 (local address 127.0.97.62:39959)
0926 23:29:01.546307 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:01.546317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:01.547952 (+ 1635us) server_negotiation.cc:407] Connection header received
0926 23:29:01.548128 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:01.548132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:01.548187 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:01.548299 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:01.549399 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:01.549403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:01.549407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:29:01.549424 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:01.549471 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:29:01.549488 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:01.549592 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:01.549734 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 23:29:01.549790 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 23:29:01.553503 22405 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1751889ec419195181ea1812b9cb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39959 } }
I20260926 23:29:01.553972 22405 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:29:01.554095 22405 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e1751889ec419195181ea1812b9cb5, State: Initialized, Role: FOLLOWER
I20260926 23:29:01.554774 22405 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1751889ec419195181ea1812b9cb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39959 } }
I20260926 23:29:01.555074 22405 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:29:01.555244 22405 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:29:01.555480 22405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:01.558081 22405 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1751889ec419195181ea1812b9cb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39959 } }
I20260926 23:29:01.558657 22405 leader_election.cc:302] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7e1751889ec419195181ea1812b9cb5; no voters:
I20260926 23:29:01.559296 22405 leader_election.cc:288] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:29:01.559442 22415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:29:01.560448 22415 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [term 1 LEADER]: Becoming Leader. State: Replica: a7e1751889ec419195181ea1812b9cb5, State: Running, Role: LEADER
I20260926 23:29:01.561113 22415 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1751889ec419195181ea1812b9cb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39959 } }
I20260926 23:29:01.561861 22405 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:01.564101 22416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7e1751889ec419195181ea1812b9cb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1751889ec419195181ea1812b9cb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39959 } } }
I20260926 23:29:01.564618 22416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:01.569751 22422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:29:01.570109 22417 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e1751889ec419195181ea1812b9cb5. Latest consensus state: current_term: 1 leader_uuid: "a7e1751889ec419195181ea1812b9cb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1751889ec419195181ea1812b9cb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39959 } } }
I20260926 23:29:01.570513 22417 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:01.575507 22422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:29:01.584983 22422 catalog_manager.cc:1347] Generated new cluster ID: f7a6249c40684c059743a36fc28590ef
I20260926 23:29:01.585103 22422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:29:01.599153 22422 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:29:01.599732 22422 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:29:01.609567 22422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7e1751889ec419195181ea1812b9cb5: Generated new TSK 0
I20260926 23:29:01.610575 22422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:29:01.619143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22338
W20260926 23:29:01.620817 22400 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52719
W20260926 23:29:01.621766 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22338: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:29:01.621951 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22338
2026-09-26T23:29:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:29:01 dist-test-slave-q5g8 krb5kdc[22436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:29:01 dist-test-slave-q5g8 krb5kdc[22436](info): set up 2 sockets
Sep 26 23:29:01 dist-test-slave-q5g8 krb5kdc[22436](info): commencing operation
krb5kdc: starting...
W20260926 23:29:02.749223 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:29:02 dist-test-slave-q5g8 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:29:03.028090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39405
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:29:03.393802 22449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:03.394140 22449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:03.394347 22449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:03.401997 22449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:03.402132 22449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:03.402261 22449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:03.402369 22449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:03.402446 22449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:03.412786 22449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:03.414233 22449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:03.415659 22449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:03.428942 22454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:03.429028 22455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:03.428997 22457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:03.429556 22449 server_base.cc:1034] running on GCE node
I20260926 23:29:03.430564 22449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:03.432199 22449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:03.433444 22449 hybrid_clock.cc:648] HybridClock initialized: now 1790465343433373 us; error 64 us; skew 500 ppm
I20260926 23:29:03.433833 22449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:03 dist-test-slave-q5g8 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:03.443050 22449 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:29:03.445031 22449 webserver.cc:460] Webserver started at http://127.0.97.62:32981/ using document root <none> and password file <none>
I20260926 23:29:03.445849 22449 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:03.446014 22449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:03.446362 22449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:03.449836 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "4a82455f6c264fb685e0b62507471082"
format_stamp: "Formatted at 2026-09-26 23:29:03 on dist-test-slave-q5g8"
server_key: "8c417f11beff76c2d39157b34219d8fe"
server_key_iv: "e1ada3043a1ca9c2ed469a4ec0c029ff"
server_key_version: "encryptionkey@0"
I20260926 23:29:03.450724 22449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "4a82455f6c264fb685e0b62507471082"
format_stamp: "Formatted at 2026-09-26 23:29:03 on dist-test-slave-q5g8"
server_key: "8c417f11beff76c2d39157b34219d8fe"
server_key_iv: "e1ada3043a1ca9c2ed469a4ec0c029ff"
server_key_version: "encryptionkey@0"
I20260926 23:29:03.456839 22449 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 23:29:03.461217 22464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:03.462960 22449 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 23:29:03.463199 22449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "4a82455f6c264fb685e0b62507471082"
format_stamp: "Formatted at 2026-09-26 23:29:03 on dist-test-slave-q5g8"
server_key: "8c417f11beff76c2d39157b34219d8fe"
server_key_iv: "e1ada3043a1ca9c2ed469a4ec0c029ff"
server_key_version: "encryptionkey@0"
I20260926 23:29:03.463436 22449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:03.492470 22449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:03.497295 22449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:03.497632 22449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:03.518060 22449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39405
I20260926 23:29:03.518107 22515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39405 every 8 connection(s)
I20260926 23:29:03.519570 22449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:29:03.523700 22516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:03.526383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22449
I20260926 23:29:03.526573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:29:03.527100 388 external_mini_cluster.cc:1428] Setting key a66b553b94d55ce8f9bb7d996833f2d4
I20260926 23:29:03.536216 22516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082: Bootstrap starting.
I20260926 23:29:03.539714 22516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:03.540930 22516 log.cc:824] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:29:03 dist-test-slave-q5g8 krb5kdc[22436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:29:03.544605 22516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082: No bootstrap required, opened a new log
I20260926 23:29:03.551054 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:03.529147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38225 (local address 127.0.97.62:39405)
0926 23:29:03.529993 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:03.530006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:03.530048 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:29:03.531757 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:03.531781 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:03.532222 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:03.532659 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:03.534190 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:03.535527 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:03.536972 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:03.537722 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:03.537947 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:03.542677 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:03.542686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:03.542706 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:03.542748 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:03.546164 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:03.547012 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:03.547020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:03.547027 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:03.547232 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:03.547692 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:03.547696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:03.547698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:03.548322 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:03.548748 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:03.549320 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 23:29:03.549816 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":363,"threads_started":1}
I20260926 23:29:03.554550 22516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a82455f6c264fb685e0b62507471082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39405 } }
I20260926 23:29:03.555104 22516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:29:03.555230 22516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a82455f6c264fb685e0b62507471082, State: Initialized, Role: FOLLOWER
I20260926 23:29:03.556083 22516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a82455f6c264fb685e0b62507471082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39405 } }
I20260926 23:29:03.556391 22516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:29:03.556576 22516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:29:03.556767 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:29:03.556792 22516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:03.560235 22516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a82455f6c264fb685e0b62507471082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39405 } }
I20260926 23:29:03.561096 22516 leader_election.cc:302] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a82455f6c264fb685e0b62507471082; no voters:
I20260926 23:29:03.561869 22516 leader_election.cc:288] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:29:03.561990 22525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:29:03.563377 22525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [term 1 LEADER]: Becoming Leader. State: Replica: 4a82455f6c264fb685e0b62507471082, State: Running, Role: LEADER
I20260926 23:29:03.564476 22516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:03.564270 22525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a82455f6c264fb685e0b62507471082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39405 } }
I20260926 23:29:03.568338 22526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a82455f6c264fb685e0b62507471082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a82455f6c264fb685e0b62507471082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39405 } } }
I20260926 23:29:03.568914 22526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:03.569661 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a82455f6c264fb685e0b62507471082. Latest consensus state: current_term: 1 leader_uuid: "4a82455f6c264fb685e0b62507471082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a82455f6c264fb685e0b62507471082" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39405 } } }
I20260926 23:29:03.570076 22534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:29:03.570076 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082 [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:03.587185 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:03.562591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38709 (local address 127.0.97.62:39405)
0926 23:29:03.562857 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:03.562866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:03.567726 (+ 4860us) server_negotiation.cc:407] Connection header received
0926 23:29:03.567992 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:03.567998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:03.568085 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:03.568204 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:03.570344 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:03.571496 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:03.573933 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:03.574957 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:03.575105 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:03.579255 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:03.579261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:03.579266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:03.579309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:03.583384 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:03.584404 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:03.584411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:03.584414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:03.584590 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:03.585100 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:03.585105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:03.585108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:03.585654 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:03.585906 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:03.586492 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 23:29:03.586762 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:29:03.590019 22534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:29:03.600775 22534 catalog_manager.cc:1347] Generated new cluster ID: e5fc203321b7414db1c4a3dad49ada54
I20260926 23:29:03.600916 22534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:29:03.613296 22534 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:29:03.614768 22534 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:29:03.631734 22534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a82455f6c264fb685e0b62507471082: Generated new TSK 0
I20260926 23:29:03.632972 22534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:29:03.645123 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22449
I20260926 23:29:03.646473 22511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:29:03.745232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22449
2026-09-26T23:29:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:29:03 dist-test-slave-q5g8 krb5kdc[22548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:29:03 dist-test-slave-q5g8 krb5kdc[22548](info): set up 2 sockets
Sep 26 23:29:03 dist-test-slave-q5g8 krb5kdc[22548](info): commencing operation
krb5kdc: starting...
W20260926 23:29:04.893280 388 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:29:05 dist-test-slave-q5g8 krb5kdc[22548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:29:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:29:05.178308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42695
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:29:05.533150 22561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:05.533500 22561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:05.533614 22561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:05.542034 22561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:05.542179 22561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:05.542228 22561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:05.542294 22561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:05.542340 22561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:05.552662 22561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:05.554061 22561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:05.555444 22561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:05.568890 22567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:05.568975 22566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:05.568931 22569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:05.569907 22561 server_base.cc:1034] running on GCE node
I20260926 23:29:05.570741 22561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:05.572232 22561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:05.573441 22561 hybrid_clock.cc:648] HybridClock initialized: now 1790465345573397 us; error 35 us; skew 500 ppm
I20260926 23:29:05.573863 22561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:05 dist-test-slave-q5g8 krb5kdc[22548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:05.577469 22561 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:29:05.579378 22561 webserver.cc:460] Webserver started at http://127.0.97.62:41679/ using document root <none> and password file <none>
I20260926 23:29:05.580210 22561 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:05.580366 22561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:05.580821 22561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:05.584628 22561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "65e11fca2f964eae9e0765bbf9b4a33c"
format_stamp: "Formatted at 2026-09-26 23:29:05 on dist-test-slave-q5g8"
server_key: "6f9990c33dde1af2b758849a0f9e4341"
server_key_iv: "0a0c6cceb49620235fd34ca514c42a31"
server_key_version: "encryptionkey@0"
I20260926 23:29:05.585600 22561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "65e11fca2f964eae9e0765bbf9b4a33c"
format_stamp: "Formatted at 2026-09-26 23:29:05 on dist-test-slave-q5g8"
server_key: "6f9990c33dde1af2b758849a0f9e4341"
server_key_iv: "0a0c6cceb49620235fd34ca514c42a31"
server_key_version: "encryptionkey@0"
I20260926 23:29:05.591348 22561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:29:05.595507 22576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:05.597175 22561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:29:05.597357 22561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "65e11fca2f964eae9e0765bbf9b4a33c"
format_stamp: "Formatted at 2026-09-26 23:29:05 on dist-test-slave-q5g8"
server_key: "6f9990c33dde1af2b758849a0f9e4341"
server_key_iv: "0a0c6cceb49620235fd34ca514c42a31"
server_key_version: "encryptionkey@0"
I20260926 23:29:05.597574 22561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:05.616452 22561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:05.621356 22561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:05.621662 22561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:05.641038 22561 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42695
I20260926 23:29:05.641072 22627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42695 every 8 connection(s)
I20260926 23:29:05.642556 22561 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:29:05.646138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22561
I20260926 23:29:05.646297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:29:05.646793 388 external_mini_cluster.cc:1428] Setting key 45b3bae917f430d89d72aeb025b4696b
I20260926 23:29:05.646867 22628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:05.664372 22628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c: Bootstrap starting.
I20260926 23:29:05.664845 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:05.648895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51529 (local address 127.0.97.62:42695)
0926 23:29:05.649813 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:05.649835 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:05.649887 (+ 52us) server_negotiation.cc:407] Connection header received
0926 23:29:05.652214 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:05.652251 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:05.652949 (+ 698us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:05.653483 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:05.654125 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:05.656376 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:05.658149 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:05.659377 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:05.659614 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:05.661492 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:05.661505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:05.661531 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:29:05.662406 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:05.662523 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:29:05.662565 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:05.662715 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:05.662940 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 23:29:05.663263 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":385,"threads_started":1}
I20260926 23:29:05.668229 22628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:05.669147 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:29:05.669602 22628 log.cc:824] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:05.672890 22628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c: No bootstrap required, opened a new log
I20260926 23:29:05.682022 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:05.674091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50379 (local address 127.0.97.62:42695)
0926 23:29:05.674369 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:05.674381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:05.675035 (+ 654us) server_negotiation.cc:407] Connection header received
0926 23:29:05.675224 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:05.675230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:05.675326 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:05.675459 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:05.676057 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:05.677277 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:05.678853 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:05.679616 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:05.679770 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:05.681056 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:05.681084 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:05.681089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:29:05.681114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:05.681175 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:29:05.681202 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:05.681317 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:05.681512 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 23:29:05.681632 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:29:05.683290 22628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42695 } }
I20260926 23:29:05.683799 22628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:29:05.683928 22628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e11fca2f964eae9e0765bbf9b4a33c, State: Initialized, Role: FOLLOWER
I20260926 23:29:05.684617 22628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42695 } }
I20260926 23:29:05.684953 22628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:29:05.685107 22628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:29:05.685254 22628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:05.688042 22628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42695 } }
I20260926 23:29:05.688598 22628 leader_election.cc:302] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65e11fca2f964eae9e0765bbf9b4a33c; no voters:
I20260926 23:29:05.689272 22628 leader_election.cc:288] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:29:05.689452 22638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:29:05.690253 22638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [term 1 LEADER]: Becoming Leader. State: Replica: 65e11fca2f964eae9e0765bbf9b4a33c, State: Running, Role: LEADER
I20260926 23:29:05.690886 22638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42695 } }
I20260926 23:29:05.691778 22628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:05.694489 22640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e11fca2f964eae9e0765bbf9b4a33c. Latest consensus state: current_term: 1 leader_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42695 } } }
I20260926 23:29:05.695055 22640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:05.696710 22644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:29:05.696534 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e11fca2f964eae9e0765bbf9b4a33c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42695 } } }
I20260926 23:29:05.696933 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:05.703241 22644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:29:05.714947 22644 catalog_manager.cc:1347] Generated new cluster ID: 1039d3b503ea46c3b5963ccff1a9bd35
I20260926 23:29:05.715091 22644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:29:05.731511 22644 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:29:05.733096 22644 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:29:05.740495 22644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65e11fca2f964eae9e0765bbf9b4a33c: Generated new TSK 0
I20260926 23:29:05.741760 22644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:29:05.753715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22561
W20260926 23:29:05.754930 22623 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51529
W20260926 23:29:05.755707 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22561: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:29:05.755885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22561
2026-09-26T23:29:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:29:05 dist-test-slave-q5g8 krb5kdc[22659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:29:05 dist-test-slave-q5g8 krb5kdc[22659](info): set up 2 sockets
Sep 26 23:29:05 dist-test-slave-q5g8 krb5kdc[22659](info): commencing operation
krb5kdc: starting...
W20260926 23:29:06.901307 388 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.011s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:29:07 dist-test-slave-q5g8 krb5kdc[22659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:29:07.180970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39791
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:29:07.546334 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:07.546669 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:07.546764 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:07.554358 22672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:07.554497 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:07.554553 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:07.554616 22672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:07.554664 22672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:07.565105 22672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:07.566478 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:07.567866 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:07.582139 22677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:07.582167 22680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:07.582247 22678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:07.586339 22672 server_base.cc:1034] running on GCE node
I20260926 23:29:07.587203 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:07.588739 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:07.589967 22672 hybrid_clock.cc:648] HybridClock initialized: now 1790465347589922 us; error 43 us; skew 500 ppm
I20260926 23:29:07.590353 22672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:07 dist-test-slave-q5g8 krb5kdc[22659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:07.593730 22672 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:29:07.595444 22672 webserver.cc:460] Webserver started at http://127.0.97.62:44635/ using document root <none> and password file <none>
I20260926 23:29:07.596196 22672 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:07.596306 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:07.596589 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:07.600030 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "577d6c2a48bb48e18324cc741be2f12c"
format_stamp: "Formatted at 2026-09-26 23:29:07 on dist-test-slave-q5g8"
server_key: "1ebcfc74598b1d09ceeb3b82c309fa49"
server_key_iv: "1aa5e70b4cb7f4f463beb32ff98736c6"
server_key_version: "encryptionkey@0"
I20260926 23:29:07.600955 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "577d6c2a48bb48e18324cc741be2f12c"
format_stamp: "Formatted at 2026-09-26 23:29:07 on dist-test-slave-q5g8"
server_key: "1ebcfc74598b1d09ceeb3b82c309fa49"
server_key_iv: "1aa5e70b4cb7f4f463beb32ff98736c6"
server_key_version: "encryptionkey@0"
I20260926 23:29:07.606889 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:29:07.611120 22687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:07.612915 22672 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:29:07.613118 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "577d6c2a48bb48e18324cc741be2f12c"
format_stamp: "Formatted at 2026-09-26 23:29:07 on dist-test-slave-q5g8"
server_key: "1ebcfc74598b1d09ceeb3b82c309fa49"
server_key_iv: "1aa5e70b4cb7f4f463beb32ff98736c6"
server_key_version: "encryptionkey@0"
I20260926 23:29:07.613351 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:07.619714 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:07.624078 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:07.624413 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:07.644651 22672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39791
I20260926 23:29:07.644783 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39791 every 8 connection(s)
I20260926 23:29:07.646216 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:29:07.649715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22672
I20260926 23:29:07.649907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:29:07.650408 388 external_mini_cluster.cc:1428] Setting key 3496d65e73a13723e4c111a8e923d063
I20260926 23:29:07.650906 22739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:07.659492 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:07.652273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34115 (local address 127.0.97.62:39791)
0926 23:29:07.653187 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:07.653200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:07.653242 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:29:07.655863 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:07.655887 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:07.656269 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:07.656627 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:07.657852 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:07.657858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:07.657871 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:29:07.657886 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:07.657956 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:29:07.657979 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:07.658159 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:07.658263 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 23:29:07.658493 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":470,"threads_started":1}
I20260926 23:29:07.663349 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:29:07.665181 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c: Bootstrap starting.
I20260926 23:29:07.669621 22739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:07.670935 22739 log.cc:824] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:07.671335 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:07.668667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41169 (local address 127.0.97.62:39791)
0926 23:29:07.668863 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:07.668870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:07.669336 (+ 466us) server_negotiation.cc:407] Connection header received
0926 23:29:07.669464 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:07.669473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:07.669562 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:07.669701 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:07.670649 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:07.670658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:07.670667 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:29:07.670690 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:07.670749 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:29:07.670773 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:07.670895 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:07.670980 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 23:29:07.671041 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:29:07.674127 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c: No bootstrap required, opened a new log
I20260926 23:29:07.682369 22739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577d6c2a48bb48e18324cc741be2f12c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260926 23:29:07.682821 22739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:29:07.682955 22739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577d6c2a48bb48e18324cc741be2f12c, State: Initialized, Role: FOLLOWER
I20260926 23:29:07.683601 22739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577d6c2a48bb48e18324cc741be2f12c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260926 23:29:07.683848 22739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:29:07.684026 22739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:29:07.684207 22739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:07.686746 22739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577d6c2a48bb48e18324cc741be2f12c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260926 23:29:07.687299 22739 leader_election.cc:302] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 577d6c2a48bb48e18324cc741be2f12c; no voters:
I20260926 23:29:07.687932 22739 leader_election.cc:288] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:29:07.688071 22749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:29:07.688877 22749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [term 1 LEADER]: Becoming Leader. State: Replica: 577d6c2a48bb48e18324cc741be2f12c, State: Running, Role: LEADER
I20260926 23:29:07.689525 22749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577d6c2a48bb48e18324cc741be2f12c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } }
I20260926 23:29:07.690522 22739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:07.692080 22750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "577d6c2a48bb48e18324cc741be2f12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577d6c2a48bb48e18324cc741be2f12c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } } }
I20260926 23:29:07.692483 22750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:07.693099 22751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 577d6c2a48bb48e18324cc741be2f12c. Latest consensus state: current_term: 1 leader_uuid: "577d6c2a48bb48e18324cc741be2f12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577d6c2a48bb48e18324cc741be2f12c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39791 } } }
I20260926 23:29:07.693385 22751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:07.693576 22755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:29:07.700114 22755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:29:07.710050 22755 catalog_manager.cc:1347] Generated new cluster ID: 6af01113dda34a19860d99c0c286a67e
I20260926 23:29:07.710186 22755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:29:07.733595 22755 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:29:07.734108 22755 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:29:07.740329 22755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 577d6c2a48bb48e18324cc741be2f12c: Generated new TSK 0
I20260926 23:29:07.741392 22755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:29:07.755131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22672
W20260926 23:29:07.757186 22734 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34115
W20260926 23:29:07.758037 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:29:07.758209 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22672
2026-09-26T23:29:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2003 ms)
[----------] 18 tests from AuthTokenIssuingTest (37183 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:29:07 dist-test-slave-q5g8 krb5kdc[22770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:29:07 dist-test-slave-q5g8 krb5kdc[22770](info): set up 2 sockets
Sep 26 23:29:07 dist-test-slave-q5g8 krb5kdc[22770](info): commencing operation
krb5kdc: starting...
W20260926 23:29:08.889356 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:29:09 dist-test-slave-q5g8 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:29:09.177145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44007
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44007,127.0.97.61:43167,127.0.97.60:41205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:29:09.533703 22783 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:29:09.534040 22783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:09.534116 22783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:09.534198 22783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:09.541877 22783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:09.541985 22783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:09.542043 22783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:09.542088 22783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:09.542201 22783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:09.552558 22783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44007,127.0.97.61:43167,127.0.97.60:41205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:09.553922 22783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:09.555235 22783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:09.567750 22791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:09.567817 22788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:09.567835 22789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:09.574275 22783 server_base.cc:1034] running on GCE node
I20260926 23:29:09.575188 22783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:09.577612 22783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:09.578933 22783 hybrid_clock.cc:648] HybridClock initialized: now 1790465349578747 us; error 134 us; skew 500 ppm
I20260926 23:29:09.579450 22783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:09 dist-test-slave-q5g8 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:09.587402 22783 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:29:09.589303 22783 webserver.cc:460] Webserver started at http://127.0.97.62:37273/ using document root <none> and password file <none>
I20260926 23:29:09.590096 22783 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:09.590204 22783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:09.590484 22783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:09.593751 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "5a5ef839dfc5477db150ba1843feda3e"
format_stamp: "Formatted at 2026-09-26 23:29:09 on dist-test-slave-q5g8"
server_key: "9de9b2cf815b7b67cbdb9e56713daaca"
server_key_iv: "712bbb36a93d83c3faaab7b6bc5ae6ec"
server_key_version: "encryptionkey@0"
I20260926 23:29:09.594537 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a5ef839dfc5477db150ba1843feda3e"
format_stamp: "Formatted at 2026-09-26 23:29:09 on dist-test-slave-q5g8"
server_key: "9de9b2cf815b7b67cbdb9e56713daaca"
server_key_iv: "712bbb36a93d83c3faaab7b6bc5ae6ec"
server_key_version: "encryptionkey@0"
I20260926 23:29:09.600466 22783 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:29:09.604907 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:09.606688 22783 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:29:09.606922 22783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "5a5ef839dfc5477db150ba1843feda3e"
format_stamp: "Formatted at 2026-09-26 23:29:09 on dist-test-slave-q5g8"
server_key: "9de9b2cf815b7b67cbdb9e56713daaca"
server_key_iv: "712bbb36a93d83c3faaab7b6bc5ae6ec"
server_key_version: "encryptionkey@0"
I20260926 23:29:09.607184 22783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:09.634435 22783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:09.639259 22783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:09.639557 22783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:09.660578 22783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44007
I20260926 23:29:09.660596 22849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44007 every 8 connection(s)
I20260926 23:29:09.662103 22783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:29:09.665026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22783
I20260926 23:29:09.665261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:29:09.665900 388 external_mini_cluster.cc:1428] Setting key b7c398e5ab71514de1f1b47c5b1780e0
I20260926 23:29:09.666203 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:09.672950 22850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:09 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 23:29:09.697148 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.676547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.0.97.62:44007)
0926 23:29:09.678361 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:09.678370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:09.678417 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:29:09.680002 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:09.680008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:09.680256 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:09.680480 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:09.681854 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:09.683051 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:09.684459 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:09.685194 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:09.685428 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:09.689186 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:09.689197 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:09.689215 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:09.689263 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:09.692077 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:09.693094 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:09.693101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:09.693104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:09.693266 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:09.693768 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:09.693771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:09.693773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:09.694311 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:09.694472 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:09.695256 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 23:29:09.695651 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1589,"thread_start_us":1350,"threads_started":1}
I20260926 23:29:09.697182 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.676544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44007 (local address 127.0.0.1:60184)
0926 23:29:09.677504 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.677556 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:09.679600 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:09.680485 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:09.680493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:09.681288 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:09.681635 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:09.681653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:09.683210 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:09.683215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:09.684313 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:09.684325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:09.685443 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:09.685450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:09.685629 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:09.686395 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:09.686418 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:09.688927 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:09.692250 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:09.692261 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:09.692274 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:09.692914 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:09.693392 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:09.693400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:09.693406 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:09.693653 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:09.694460 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:09.694472 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:09.694791 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 23:29:09.695079 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 23:29:09.695845 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":200,"thread_start_us":484,"threads_started":1}
I20260926 23:29:09.700538 22850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:29:09.701709 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.701296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57638)
0926 23:29:09.701475 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.701586 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 23:29:09.702097 22802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
I20260926 23:29:09.703835 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.703418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57650)
0926 23:29:09.703590 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.703704 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 23:29:09.704106 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 23:29:09.729058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43167
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44007,127.0.97.61:43167,127.0.97.60:41205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:29:09.753508 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } attempt: 1
I20260926 23:29:09.754781 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.754277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57656)
0926 23:29:09.754509 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.754620 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 23:29:09.756208 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.755662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57670)
0926 23:29:09.755947 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.756040 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260926 23:29:09.756414 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
I20260926 23:29:09.824743 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } attempt: 2
I20260926 23:29:09.826351 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.825707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57678)
0926 23:29:09.826007 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.826159 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:29:09.827692 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.827212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57684)
0926 23:29:09.827424 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.827519 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260926 23:29:09.827937 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
I20260926 23:29:09.919265 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } attempt: 3
I20260926 23:29:09.920666 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.920046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57690)
0926 23:29:09.920317 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.920450 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 23:29:09.921950 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:09.921437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57704)
0926 23:29:09.921635 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:09.921741 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 23:29:09.922211 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
I20260926 23:29:10.065479 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } attempt: 4
I20260926 23:29:10.066958 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.066310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57712)
0926 23:29:10.066719 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.066817 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
I20260926 23:29:10.068207 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.067807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57718)
0926 23:29:10.067970 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.068069 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 23:29:10.068476 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43167: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43167: connect: Connection refused (error 111)
W20260926 23:29:10.102602 22857 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:29:10.103013 22857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:10.103160 22857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:10.103258 22857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:10.111289 22857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:10.111461 22857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:10.111613 22857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:10.111681 22857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:10.111743 22857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:10.122540 22857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44007,127.0.97.61:43167,127.0.97.60:41205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:10.123976 22857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:10.125464 22857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:10.138700 22862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:10.138773 22863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:10.138748 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:10.140168 22857 server_base.cc:1034] running on GCE node
I20260926 23:29:10.141315 22857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:10.143152 22857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:10.144470 22857 hybrid_clock.cc:648] HybridClock initialized: now 1790465350144399 us; error 48 us; skew 500 ppm
I20260926 23:29:10.145010 22857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:10.148998 22857 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 23:29:10.150739 22857 webserver.cc:460] Webserver started at http://127.0.97.61:43353/ using document root <none> and password file <none>
I20260926 23:29:10.151556 22857 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:10.151716 22857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:10.152067 22857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:10.155467 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data/instance:
uuid: "642c25edd86241db80f0c7368942ec3a"
format_stamp: "Formatted at 2026-09-26 23:29:10 on dist-test-slave-q5g8"
server_key: "5fc0fc72236b143928af26466fa3f1f0"
server_key_iv: "eafc4c25d5a3c76a37cfc611775cb1ee"
server_key_version: "encryptionkey@0"
I20260926 23:29:10.156369 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/wal/instance:
uuid: "642c25edd86241db80f0c7368942ec3a"
format_stamp: "Formatted at 2026-09-26 23:29:10 on dist-test-slave-q5g8"
server_key: "5fc0fc72236b143928af26466fa3f1f0"
server_key_iv: "eafc4c25d5a3c76a37cfc611775cb1ee"
server_key_version: "encryptionkey@0"
I20260926 23:29:10.162325 22857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 23:29:10.166522 22872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:10.168075 22857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260926 23:29:10.168303 22857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/wal
uuid: "642c25edd86241db80f0c7368942ec3a"
format_stamp: "Formatted at 2026-09-26 23:29:10 on dist-test-slave-q5g8"
server_key: "5fc0fc72236b143928af26466fa3f1f0"
server_key_iv: "eafc4c25d5a3c76a37cfc611775cb1ee"
server_key_version: "encryptionkey@0"
I20260926 23:29:10.168546 22857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:10.203082 22857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:10.207753 22857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:10.208083 22857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:10.228052 22857 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43167
I20260926 23:29:10.228060 22923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43167 every 8 connection(s)
I20260926 23:29:10.229720 22857 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:29:10.234493 22924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:10.238830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22857
I20260926 23:29:10.239084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-1/wal/instance
I20260926 23:29:10.239564 388 external_mini_cluster.cc:1428] Setting key 75ead65809413e1302850c6c4589dbda
I20260926 23:29:10.242481 22924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 23:29:10.267314 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.245429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.0.97.62:44007)
0926 23:29:10.246023 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.246029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.247214 (+ 1185us) server_negotiation.cc:407] Connection header received
0926 23:29:10.249573 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.249582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.249740 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.249874 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.251607 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.252884 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.254493 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.255283 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.255439 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.259524 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.259535 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.259544 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.259612 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.263886 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.264817 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.264824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.264829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.264989 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.265480 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.265486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.265492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.265930 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.266084 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.266855 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.267034 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":302,"threads_started":1}
I20260926 23:29:10.268344 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.246048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44007 (local address 127.0.0.1:60194)
0926 23:29:10.246988 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.247047 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:10.249239 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:10.250102 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:10.250110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:10.250684 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:10.251339 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.251358 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.253051 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.253055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.254320 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.254334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.255510 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.255515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.255732 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.256503 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:10.256528 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:10.259222 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:10.264081 (+ 4859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.264090 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.264107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.264624 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.265132 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.265135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.265138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.265340 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.266054 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:10.266061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:10.266482 (+ 421us) client_negotiation.cc:769] Sending connection context
0926 23:29:10.266735 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 23:29:10.267151 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":338,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 23:29:10.270550 22924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:29:10.287977 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.271248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57722)
0926 23:29:10.271551 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.271573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:10.271698 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:10.272930 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:10.272938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:10.273531 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:10.273764 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.273777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.275193 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.275201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.276735 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.276752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.277650 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.277662 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.277880 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.278598 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:10.278618 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:10.280941 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:10.285456 (+ 4515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.285460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.285463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.285871 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.286321 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.286324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.286326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.286455 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.287255 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:10.287259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:10.287374 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 23:29:10.287486 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 23:29:10.287710 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 23:29:10.288753 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.271609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.97.61:43167)
0926 23:29:10.272385 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.272390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.272418 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:29:10.272465 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.272471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.272771 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.272895 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.273959 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.275027 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.276874 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.277445 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.277546 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.281178 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.281196 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.281231 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.281275 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.285298 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.286017 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.286030 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.286036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.286203 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.286562 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.286570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.286576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.287104 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.287329 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.287698 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.288002 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":412,"threads_started":1}
I20260926 23:29:10.291338 22924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:29:10.292769 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.292196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47458)
0926 23:29:10.292390 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.292538 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 23:29:10.293072 22874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.294998 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.294472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47470)
0926 23:29:10.294695 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.294822 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 23:29:10.295308 22924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 23:29:10.308499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41205
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44007,127.0.97.61:43167,127.0.97.60:41205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:29:10.344756 22924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 1
I20260926 23:29:10.346082 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.345508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47472)
0926 23:29:10.345753 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.345873 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:29:10.347245 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.346814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47480)
0926 23:29:10.346978 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.347092 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:29:10.347478 22924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.367836 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } attempt: 5
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:29:10.383937 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.368723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57726)
0926 23:29:10.368930 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.368955 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:10.369076 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:10.369535 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:10.369539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:10.369936 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:10.370141 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.370152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.371765 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.371769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.372828 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.372843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.373946 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.373951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.374142 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.374853 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:10.374874 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:10.377229 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:10.381239 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.381243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.381246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.381759 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.382226 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.382229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.382232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.382369 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.383097 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:10.383100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:10.383258 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 23:29:10.383412 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 23:29:10.383662 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 23:29:10.384186 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.368824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.0.97.61:43167)
0926 23:29:10.369075 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.369084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.369116 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:29:10.369203 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.369212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.369356 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.369516 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.370344 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.371576 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.372991 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.373776 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.373914 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.377481 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.377490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.377495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.377542 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.381053 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.381921 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.381929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.381932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.382105 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.382489 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.382493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.382496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.382967 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.383139 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.383578 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.383851 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 23:29:10.385426 22850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:29:10.386698 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.386216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47496)
0926 23:29:10.386418 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.386524 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 23:29:10.387959 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.387470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47504)
0926 23:29:10.387721 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.387805 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260926 23:29:10.388223 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.415743 22924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 2
I20260926 23:29:10.417140 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.416521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47506)
0926 23:29:10.416798 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.416923 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:29:10.418322 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.417892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47520)
0926 23:29:10.418065 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.418164 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:29:10.418582 22924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.439510 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 1
I20260926 23:29:10.440877 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.440297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47528)
0926 23:29:10.440509 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.440607 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 23:29:10.441926 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.441580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47534)
0926 23:29:10.441751 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.441822 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:29:10.442137 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.509891 22924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 3
I20260926 23:29:10.510442 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 2
I20260926 23:29:10.511633 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.510888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47540)
0926 23:29:10.511210 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.511355 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 23:29:10.511672 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.511064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47550)
0926 23:29:10.511316 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.511423 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:29:10.513033 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.512413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47558)
0926 23:29:10.512732 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.512862 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 23:29:10.513216 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.512811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47568)
0926 23:29:10.512965 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.513060 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 23:29:10.513341 22924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
W20260926 23:29:10.513418 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.619688 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 3
I20260926 23:29:10.621119 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.620428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47578)
0926 23:29:10.620752 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.620894 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 23:29:10.622457 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.621903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47586)
0926 23:29:10.622165 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.622286 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260926 23:29:10.622761 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.656635 22924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 4
I20260926 23:29:10.658116 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.657443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47594)
0926 23:29:10.657842 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.657935 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260926 23:29:10.659264 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.658856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47608)
0926 23:29:10.659043 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.659142 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 23:29:10.659500 22924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
W20260926 23:29:10.692929 22932 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:29:10.693375 22932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:10.693516 22932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:10.693684 22932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:10.702709 22932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:10.702893 22932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:10.703042 22932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:10.703147 22932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:10.703279 22932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:10.715039 22932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44007,127.0.97.61:43167,127.0.97.60:41205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:10.716614 22932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:10.718327 22932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:10.732585 22938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:10.732504 22940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:10.732473 22937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:10.733557 22932 server_base.cc:1034] running on GCE node
I20260926 23:29:10.734431 22932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:10.735978 22932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:10.737246 22932 hybrid_clock.cc:648] HybridClock initialized: now 1790465350737168 us; error 42 us; skew 500 ppm
I20260926 23:29:10.737711 22932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:10.741704 22932 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 23:29:10.743494 22932 webserver.cc:460] Webserver started at http://127.0.97.60:44319/ using document root <none> and password file <none>
I20260926 23:29:10.744308 22932 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:10.744457 22932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:10.744882 22932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:10.748293 22932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data/instance:
uuid: "1a192f520e114ec7b404ef9fec7e5c95"
format_stamp: "Formatted at 2026-09-26 23:29:10 on dist-test-slave-q5g8"
server_key: "e3323e16e789243f7e551e6139249bdc"
server_key_iv: "4e96fa1e9793e83bd4329e34902653c8"
server_key_version: "encryptionkey@0"
I20260926 23:29:10.749472 22932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/wal/instance:
uuid: "1a192f520e114ec7b404ef9fec7e5c95"
format_stamp: "Formatted at 2026-09-26 23:29:10 on dist-test-slave-q5g8"
server_key: "e3323e16e789243f7e551e6139249bdc"
server_key_iv: "4e96fa1e9793e83bd4329e34902653c8"
server_key_version: "encryptionkey@0"
I20260926 23:29:10.755425 22932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:29:10.759969 22947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:10.762053 22932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:29:10.762287 22932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/wal
uuid: "1a192f520e114ec7b404ef9fec7e5c95"
format_stamp: "Formatted at 2026-09-26 23:29:10 on dist-test-slave-q5g8"
server_key: "e3323e16e789243f7e551e6139249bdc"
server_key_iv: "4e96fa1e9793e83bd4329e34902653c8"
server_key_version: "encryptionkey@0"
I20260926 23:29:10.762564 22932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:10.777992 22932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:10.783679 22932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:10.784017 22932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:10.800007 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 4
I20260926 23:29:10.801620 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.800917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47620)
0926 23:29:10.801239 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.801380 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 23:29:10.803155 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.802551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47624)
0926 23:29:10.802773 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.802908 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 23:29:10.803483 22850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41205: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41205: connect: Connection refused (error 111)
I20260926 23:29:10.806984 22932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41205
I20260926 23:29:10.806991 22998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41205 every 8 connection(s)
I20260926 23:29:10.808619 22932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:29:10.810302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 22932
I20260926 23:29:10.810498 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790464978839002-388-0/minicluster-data/master-2/wal/instance
I20260926 23:29:10.811056 388 external_mini_cluster.cc:1428] Setting key c918143ccda30e15547f344b130eb1f6
I20260926 23:29:10.813858 22999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:10.823021 22999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:29:10.838069 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.813266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.0.97.60:41205)
0926 23:29:10.814091 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.814110 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.814154 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:29:10.815845 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.815873 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.816317 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.816801 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.818615 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.820036 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.822025 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.822870 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.823033 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.829042 (+ 6009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.829055 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.829077 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.829134 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.833062 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.833979 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.833987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.833994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.834182 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.834645 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.834649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.834652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.835523 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.835871 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.836288 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.836849 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"spinlock_wait_cycles":37248,"thread_start_us":382,"threads_started":1}
I20260926 23:29:10.845058 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.827023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.0.97.62:44007)
0926 23:29:10.827715 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.827726 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.828655 (+ 929us) server_negotiation.cc:407] Connection header received
0926 23:29:10.828960 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.828968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.829079 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.829218 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.830738 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.832033 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.833670 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.834256 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.834409 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.838125 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.838130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.838134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.838170 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.841501 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.842509 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.842515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.842520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.842754 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.843138 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.843143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.843148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.843540 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.843733 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.844397 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.844580 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":322,"threads_started":1}
I20260926 23:29:10.845054 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.827030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44007 (local address 127.0.0.1:60206)
0926 23:29:10.828463 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.828522 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:10.828745 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:10.829270 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:10.829277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:10.830199 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:10.830538 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.830555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.832225 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.832229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.833496 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.833509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.834428 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.834433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.834601 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.835255 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:10.835278 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:10.837906 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:10.841722 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.841737 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.841744 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.842358 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.842878 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.842882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.842884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.843011 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.843724 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:10.843729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:10.843999 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 23:29:10.844220 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 23:29:10.844569 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"thread_start_us":960,"threads_started":1}
I20260926 23:29:10.846696 22999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:29:10.859630 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.841315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.97.62:44007)
0926 23:29:10.841984 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.841990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.842013 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:29:10.842066 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.842071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.842149 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.842278 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.843867 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.845344 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.847211 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.847880 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.848005 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.853082 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.853087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.853091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.853130 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.856268 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.857304 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.857311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.857315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.857485 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.857971 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.857977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.857983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.858300 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.858648 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.858945 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.859210 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":294,"threads_started":1}
I20260926 23:29:10.863883 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.847470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43167 (local address 127.0.0.1:57734)
0926 23:29:10.847656 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.847677 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:10.847780 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:10.848190 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:10.848195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:10.848738 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:10.848949 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.848962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.850064 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.850073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.851558 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.851572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.852547 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.852556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.852824 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.853505 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:10.853523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:10.856072 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:10.860145 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.860153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.860161 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.860807 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.861484 (+ 677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.861490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.861494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.861770 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.862739 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:10.862744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:10.862952 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 23:29:10.863126 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 23:29:10.863520 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 23:29:10.864123 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.847607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.97.61:43167)
0926 23:29:10.847827 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.847834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.847857 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:29:10.847921 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.847930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.848022 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.848128 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.849118 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.849875 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.851734 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.852359 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.852471 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.856388 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.856396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.856403 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.856455 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.859917 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.860967 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.860978 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.860982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.861261 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.861958 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.861965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.861969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.862413 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.862644 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.863316 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.863654 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:29:10.865208 22999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:29:10.881108 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.866173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.97.60:41205)
0926 23:29:10.866385 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.866391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.866556 (+ 165us) server_negotiation.cc:407] Connection header received
0926 23:29:10.866651 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.866656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.866741 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.866859 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.867850 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.868964 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.870693 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.871105 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.871205 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.875053 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.875059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.875063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.875097 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.878230 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.879010 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.879016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.879026 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.879171 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.879564 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.879570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.879575 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.879862 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.880026 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.880365 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.880630 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:29:10.881364 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.866212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47650)
0926 23:29:10.866454 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.866470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:10.866566 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:10.866913 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:10.866917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:10.867505 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:10.867718 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.867731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.869129 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.869133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.870516 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.870535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.871305 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.871313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.871588 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.872346 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:10.872378 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:10.874842 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:10.878419 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.878425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.878429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.878869 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.879287 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.879294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.879300 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.879441 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.879985 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:10.879990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:10.880140 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 23:29:10.880288 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 23:29:10.880872 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":171}
I20260926 23:29:10.881639 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.862127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.0.97.61:43167)
0926 23:29:10.862859 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.862865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.862889 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:29:10.862941 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.862946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.863023 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.863170 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.865560 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.866820 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.868493 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.869216 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.869314 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.875033 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.875040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.875044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.875088 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.878400 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.879215 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.879223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.879227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.879399 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.879891 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.879896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.879900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.880231 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.880527 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.880908 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.881172 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":332,"threads_started":1}
I20260926 23:29:10.887020 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:29:10.887483 22999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: Bootstrap starting.
I20260926 23:29:10.891075 22999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:10.892252 22999 log.cc:824] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:10.889863 22825 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5a5ef839dfc5477db150ba1843feda3e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60222
I20260926 23:29:10.895429 22999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: No bootstrap required, opened a new log
I20260926 23:29:10.903637 22999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
I20260926 23:29:10.904030 22999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a192f520e114ec7b404ef9fec7e5c95, State: Initialized, Role: FOLLOWER
I20260926 23:29:10.904577 22999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
I20260926 23:29:10.905738 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:10.906147 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:10.906831 22999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:29:10.917054 23019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:29:10.917200 23019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:29:10.914912 22932 master_runner.cc:186] Error getting master registration for 127.0.97.62:44007: OK, instance_id { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" instance_seqno: 1790465349654614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:29:10.927946 22932 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:29:10.958753 22924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 5
Sep 26 23:29:10 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:29:10.977164 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.959584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47654)
0926 23:29:10.959852 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:10.959874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:10.960019 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:10.960512 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:10.960534 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:10.961136 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:10.961342 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.961353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.962961 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.962970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.964533 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:10.964553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.965733 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.965738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:10.965911 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.966615 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:10.966634 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:10.969248 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:10.973794 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.973800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.973805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.974428 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.974936 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:10.974945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:10.974951 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:10.975136 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:10.975961 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:10.975966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:10.976159 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 23:29:10.976328 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 23:29:10.976708 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 23:29:10.977304 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:10.959857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.0.97.60:41205)
0926 23:29:10.960107 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:10.960114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:10.960138 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:29:10.960210 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:10.960216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:10.960302 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:10.960433 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:10.961515 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.962794 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.964724 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:10.965499 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:10.965701 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:10.969474 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:10.969481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:10.969486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:10.969540 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:10.973576 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.974594 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.974601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.974604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.974790 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:10.975259 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:10.975265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:10.975269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:10.975816 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:10.976009 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:10.976461 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 23:29:10.976888 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:29:10.984104 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: Bootstrap starting.
I20260926 23:29:10.988581 22924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:10.990124 22924 log.cc:824] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:10.994664 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: No bootstrap required, opened a new log
I20260926 23:29:11.007236 22924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
I20260926 23:29:11.007745 22924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642c25edd86241db80f0c7368942ec3a, State: Initialized, Role: FOLLOWER
I20260926 23:29:11.008363 22924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
I20260926 23:29:11.009557 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.010016 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:11.011178 22924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:29:11.023770 23033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:29:11.024017 23033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:29:11.020195 22857 master_runner.cc:186] Error getting master registration for 127.0.97.62:44007: OK, instance_id { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" instance_seqno: 1790465349654614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:29:11.033182 22857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 23:29:11.036769 22953 tablet.cc:2369] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:29:11.080765 22850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } attempt: 5
Sep 26 23:29:11 dist-test-slave-q5g8 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:29:11.099651 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:11.082345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.0.97.60:41205)
0926 23:29:11.082523 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.082528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.082545 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:29:11.082600 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.082604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.082668 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.082805 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.083850 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.084647 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.086578 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.087344 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.087490 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:11.091823 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:11.091829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:11.091833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:11.091875 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:11.096087 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.097051 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.097060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.097064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.097272 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.097855 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.097861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.097865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.098213 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:11.098400 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:11.098884 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 23:29:11.099202 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:29:11.099669 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:11.081583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41205 (local address 127.0.0.1:47666)
0926 23:29:11.081909 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:11.081931 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:11.082074 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:11.082843 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:11.082848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:11.083405 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:11.083647 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:11.083663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.084844 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.084851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:11.086362 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:11.086382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.087524 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.087531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:11.087756 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:11.088465 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:11.088486 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:11.091562 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:11.096274 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:11.096280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:11.096285 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:11.096884 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:11.097440 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:11.097447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:11.097454 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:11.097729 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:11.098364 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:11.098369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:11.098579 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 23:29:11.098760 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 23:29:11.099171 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260926 23:29:11.105628 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e: Bootstrap starting.
I20260926 23:29:11.110471 22850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:11.112167 22850 log.cc:824] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:11.116765 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e: No bootstrap required, opened a new log
I20260926 23:29:11.125783 22850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
I20260926 23:29:11.126710 22850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5ef839dfc5477db150ba1843feda3e, State: Initialized, Role: FOLLOWER
I20260926 23:29:11.127292 22850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
I20260926 23:29:11.134392 23036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.134841 23036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:11.136828 22850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:11.136879 22825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 23:29:11.137183 22825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:11.141762 22825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
W20260926 23:29:11.143142 22804 tablet.cc:2369] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:29:11.146824 22899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a5ef839dfc5477db150ba1843feda3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "642c25edd86241db80f0c7368942ec3a"
I20260926 23:29:11.147591 22899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:11.153357 22899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a5ef839dfc5477db150ba1843feda3e in term 1.
I20260926 23:29:11.154564 22802 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5ef839dfc5477db150ba1843feda3e, 642c25edd86241db80f0c7368942ec3a; no voters:
I20260926 23:29:11.155205 23036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:29:11.156847 22974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a5ef839dfc5477db150ba1843feda3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1a192f520e114ec7b404ef9fec7e5c95"
I20260926 23:29:11.157606 22974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:11.162933 22974 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a5ef839dfc5477db150ba1843feda3e in term 1.
I20260926 23:29:11.163936 22825 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [CANDIDATE]: Term 1 election: Requested vote from peers 642c25edd86241db80f0c7368942ec3a (127.0.97.61:43167), 1a192f520e114ec7b404ef9fec7e5c95 (127.0.97.60:41205)
I20260926 23:29:11.169835 23036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [term 1 LEADER]: Becoming Leader. State: Replica: 5a5ef839dfc5477db150ba1843feda3e, State: Running, Role: LEADER
I20260926 23:29:11.170524 23036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } }
I20260926 23:29:11.179008 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5ef839dfc5477db150ba1843feda3e. Latest consensus state: current_term: 1 leader_uuid: "5a5ef839dfc5477db150ba1843feda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.179584 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:11.191413 23044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 23:29:11.211114 22878 tablet.cc:2369] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:29:11.211436 23044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:29:11.233735 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:11.193690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.0.97.60:41205)
0926 23:29:11.193925 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.193934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.202374 (+ 8440us) server_negotiation.cc:407] Connection header received
0926 23:29:11.204583 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.204590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.204723 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.204815 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.213200 (+ 8385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.214268 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.216801 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.217542 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.217648 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:11.223972 (+ 6324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:11.223981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:11.223991 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:11.224055 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:11.228318 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.230227 (+ 1909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.230233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.230237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.230423 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.231291 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.231296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.231300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.231677 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:11.231952 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:11.233001 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0926 23:29:11.233320 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:29:11.249600 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:11.194710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.0.97.62:44007)
0926 23:29:11.194907 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.194914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.209657 (+ 14743us) server_negotiation.cc:407] Connection header received
0926 23:29:11.210016 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.210023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.210140 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.210217 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.212503 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.213710 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.215808 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.216567 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.216675 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:11.232864 (+ 16189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:11.232872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:11.232878 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:11.232939 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:11.237044 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.245893 (+ 8849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.245900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.245905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.246131 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.246660 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.246666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.246671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.247139 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:11.247431 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:11.248859 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0926 23:29:11.249189 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 23:29:11.252926 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:11.209870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.0.97.61:43167)
0926 23:29:11.210050 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.210055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.215339 (+ 5284us) server_negotiation.cc:407] Connection header received
0926 23:29:11.217454 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.217461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.217555 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.217626 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.225560 (+ 7934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.226834 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.232912 (+ 6078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:11.234193 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:11.234340 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:11.240528 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:11.240536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:11.240542 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:11.240595 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:11.244649 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.247988 (+ 3339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.247995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.247998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.248198 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:11.250691 (+ 2493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:11.250698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:11.250702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:11.251162 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:11.251457 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:11.252204 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 23:29:11.252505 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 23:29:11.252889 22974 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [term 1 FOLLOWER]: Refusing update from remote peer 5a5ef839dfc5477db150ba1843feda3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:29:11.252991 23059 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:29:11.254418 23038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:29:11.256937 22899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [term 1 FOLLOWER]: Refusing update from remote peer 5a5ef839dfc5477db150ba1843feda3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:29:11.264273 23038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:29:11.285147 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5ef839dfc5477db150ba1843feda3e. Latest consensus state: current_term: 1 leader_uuid: "5a5ef839dfc5477db150ba1843feda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.285650 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:11.294206 23036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a5ef839dfc5477db150ba1843feda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.294592 23036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:11.297294 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5ef839dfc5477db150ba1843feda3e. Latest consensus state: current_term: 1 leader_uuid: "5a5ef839dfc5477db150ba1843feda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.297668 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:11.299779 23044 catalog_manager.cc:1347] Generated new cluster ID: 84794cf1c1c14b88abb2e126cdd568de
I20260926 23:29:11.299904 23044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:29:11.303668 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a5ef839dfc5477db150ba1843feda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.303954 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:11.308401 23060 mvcc.cc:205] Tried to move back new op lower bound from 7333746078647656448 to 7333746078400802816. Current Snapshot: MvccSnapshot[applied={T|T < 7333746078647656448}]
I20260926 23:29:11.312264 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a5ef839dfc5477db150ba1843feda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.312553 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:11.329499 23008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a5ef839dfc5477db150ba1843feda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5ef839dfc5477db150ba1843feda3e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44007 } } peers { permanent_uuid: "642c25edd86241db80f0c7368942ec3a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43167 } } peers { permanent_uuid: "1a192f520e114ec7b404ef9fec7e5c95" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41205 } } }
I20260926 23:29:11.329756 23008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:11.336388 23044 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:29:11.343915 23044 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:29:11.366719 23044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5ef839dfc5477db150ba1843feda3e: Generated new TSK 0
I20260926 23:29:11.368167 23044 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:29:11.506428 23002 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.491707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.0.97.60:41205)
0926 23:29:11.491922 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.491931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.503884 (+ 11953us) server_negotiation.cc:407] Connection header received
0926 23:29:11.504491 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.504499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.504602 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.504684 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.506110 (+ 1426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47688: BlockingRecv error: recv got EOF from 127.0.0.1:47688 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 23:29:11.510749 23005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.499765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.0.97.62:44007)
0926 23:29:11.500035 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.500043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.507634 (+ 7591us) server_negotiation.cc:407] Connection header received
0926 23:29:11.507973 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.507981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.508102 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.508225 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.510385 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60232: BlockingRecv error: recv got EOF from 127.0.0.1:60232 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 23:29:11.515548 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.512511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.97.61:43167)
0926 23:29:11.512817 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.512825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.513420 (+ 595us) server_negotiation.cc:407] Connection header received
0926 23:29:11.513716 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.513724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.513817 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.513902 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.515233 (+ 1331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57760: BlockingRecv error: recv got EOF from 127.0.0.1:57760 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 23:29:11.557247 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.547842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.0.97.61:43167)
0926 23:29:11.548052 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.548059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.555075 (+ 7016us) server_negotiation.cc:407] Connection header received
0926 23:29:11.555673 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.555686 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.555861 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.556017 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.556959 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57772: BlockingRecv error: recv got EOF from 127.0.0.1:57772 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 23:29:11.566974 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.561982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.0.97.61:43167)
0926 23:29:11.562226 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.562232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.565287 (+ 3055us) server_negotiation.cc:407] Connection header received
0926 23:29:11.565527 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.565534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.565630 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.565741 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.566653 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57782: BlockingRecv error: recv got EOF from 127.0.0.1:57782 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 23:29:11.575381 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.573147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.0.97.61:43167)
0926 23:29:11.573349 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.573356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.573908 (+ 552us) server_negotiation.cc:407] Connection header received
0926 23:29:11.574106 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.574113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.574215 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.574330 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.575137 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57792: BlockingRecv error: recv got EOF from 127.0.0.1:57792 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 23:29:11.587353 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.583915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.97.61:43167)
0926 23:29:11.584179 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.584186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.585365 (+ 1179us) server_negotiation.cc:407] Connection header received
0926 23:29:11.585711 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.585718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.585832 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.585921 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.587050 (+ 1129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57800: BlockingRecv error: recv got EOF from 127.0.0.1:57800 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260926 23:29:11.608955 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.606379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.0.97.61:43167)
0926 23:29:11.606565 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.606572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.607213 (+ 641us) server_negotiation.cc:407] Connection header received
0926 23:29:11.607425 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.607432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.607567 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.607688 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.608673 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57802: BlockingRecv error: recv got EOF from 127.0.0.1:57802 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 23:29:11.633196 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.630754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.0.97.61:43167)
0926 23:29:11.630991 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.630997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.631557 (+ 560us) server_negotiation.cc:407] Connection header received
0926 23:29:11.631761 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.631767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.631869 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.631982 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.632883 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57808: BlockingRecv error: recv got EOF from 127.0.0.1:57808 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 23:29:11.673986 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.668626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.97.61:43167)
0926 23:29:11.668858 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.668865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.672438 (+ 3573us) server_negotiation.cc:407] Connection header received
0926 23:29:11.672726 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.672734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.672845 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.672952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.673767 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57818: BlockingRecv error: recv got EOF from 127.0.0.1:57818 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 23:29:11.744269 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.741702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.0.97.61:43167)
0926 23:29:11.741955 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.741963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.742501 (+ 538us) server_negotiation.cc:407] Connection header received
0926 23:29:11.742697 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.742704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.742828 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.742962 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.743986 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57826: BlockingRecv error: recv got EOF from 127.0.0.1:57826 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 23:29:11.878249 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:11.876215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.0.97.61:43167)
0926 23:29:11.876423 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:11.876428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:11.876924 (+ 496us) server_negotiation.cc:407] Connection header received
0926 23:29:11.877151 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:11.877156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:11.877232 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:11.877335 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:11.878061 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57830: BlockingRecv error: recv got EOF from 127.0.0.1:57830 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:29:11.919229 23019 catalog_manager.cc:1259] Loaded cluster ID: 84794cf1c1c14b88abb2e126cdd568de
I20260926 23:29:11.919334 23019 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: loading cluster ID for follower catalog manager: success
I20260926 23:29:11.921903 23019 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: acquiring CA information for follower catalog manager: success
I20260926 23:29:11.923731 23019 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a192f520e114ec7b404ef9fec7e5c95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:29:12.026145 23033 catalog_manager.cc:1259] Loaded cluster ID: 84794cf1c1c14b88abb2e126cdd568de
I20260926 23:29:12.026247 23033 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: loading cluster ID for follower catalog manager: success
I20260926 23:29:12.028600 23033 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: acquiring CA information for follower catalog manager: success
I20260926 23:29:12.030442 23033 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 642c25edd86241db80f0c7368942ec3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:29:12.145691 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:12.139585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.97.61:43167)
0926 23:29:12.139809 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:12.139816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:12.139983 (+ 167us) server_negotiation.cc:407] Connection header received
0926 23:29:12.140227 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:12.140233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:12.140398 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:12.140526 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:12.141020 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.141817 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.143269 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.143699 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.143841 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:12.144385 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:12.145042 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:12.145223 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:12.145279 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 23:29:12.145379 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:29:12.158469 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:12.152356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.0.97.60:41205)
0926 23:29:12.152908 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:12.152914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:12.152940 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:29:12.153174 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:12.153181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:12.153346 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:12.153517 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:12.153986 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.154797 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.156145 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.156549 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.156651 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:12.157267 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:12.157916 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:12.158070 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:12.158116 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 23:29:12.158194 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":342,"threads_started":1}
I20260926 23:29:12.176262 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:12.167170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.0.97.60:41205)
0926 23:29:12.167364 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:12.167373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:12.168737 (+ 1364us) server_negotiation.cc:407] Connection header received
0926 23:29:12.169132 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:12.169142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:12.169273 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:12.169397 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:12.170027 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.171508 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.174024 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.174726 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.174865 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:12.175436 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:12.175567 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:12.175662 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:12.175837 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 23:29:12.175925 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 23:29:12.179816 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:12.166546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.97.62:44007)
0926 23:29:12.173612 (+ 7066us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:12.173620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:12.173640 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:29:12.173771 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:12.173777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:12.174020 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:12.174232 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:12.174823 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.175674 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.177458 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.178078 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.178204 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:12.178642 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:12.179140 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:12.179235 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:12.179395 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 23:29:12.179488 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6924,"thread_start_us":6818,"threads_started":1}
I20260926 23:29:12.179888 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:12.167345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.0.97.61:43167)
0926 23:29:12.167574 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:12.167582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:12.173557 (+ 5975us) server_negotiation.cc:407] Connection header received
0926 23:29:12.173825 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:12.173831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:12.173914 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:12.174018 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:12.174927 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.175901 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.177857 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:12.178493 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:12.178598 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:12.179162 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:12.179308 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:12.179394 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:12.179569 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 23:29:12.179665 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:29:12.185127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22783
I20260926 23:29:12.186192 22845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:29:12.305860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22783
I20260926 23:29:12.330343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22857
W20260926 23:29:12.330760 22876 connection.cc:535] server connection from 127.0.0.1:57726 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:29:12.331727 22919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:29:12.441430 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22857
I20260926 23:29:12.461977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 22932
I20260926 23:29:12.463171 22994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:29:12.573554 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 22932
2026-09-26T23:29:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4820 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:29:12 dist-test-slave-q5g8 krb5kdc[23146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:29:12 dist-test-slave-q5g8 krb5kdc[23146](info): set up 2 sockets
Sep 26 23:29:12 dist-test-slave-q5g8 krb5kdc[23146](info): commencing operation
krb5kdc: starting...
W20260926 23:29:13.709422 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/test-user.keytab.
Sep 26 23:29:13 dist-test-slave-q5g8 krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:29:13.991791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46177
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46177,127.0.97.61:43725,127.0.97.60:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:29:14.368135 23159 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:29:14.368587 23159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:14.368748 23159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:14.368933 23159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:14.376916 23159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:14.377068 23159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:14.377194 23159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:14.377297 23159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:14.377404 23159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:14.389191 23159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46177,127.0.97.61:43725,127.0.97.60:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:14.390810 23159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:14.392369 23159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:14.405632 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:14.405653 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:14.405611 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:14.406618 23159 server_base.cc:1034] running on GCE node
I20260926 23:29:14.407491 23159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:14.409286 23159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:14.410583 23159 hybrid_clock.cc:648] HybridClock initialized: now 1790465354410515 us; error 51 us; skew 500 ppm
I20260926 23:29:14.410982 23159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:14 dist-test-slave-q5g8 krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:14.414708 23159 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:29:14.416486 23159 webserver.cc:460] Webserver started at http://127.0.97.62:43239/ using document root <none> and password file <none>
I20260926 23:29:14.417389 23159 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:14.417551 23159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:14.417927 23159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:14.421383 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data/instance:
uuid: "5f906dc8bec64e7eaba43a1777831684"
format_stamp: "Formatted at 2026-09-26 23:29:14 on dist-test-slave-q5g8"
server_key: "d79ae8eb98889a1220ddcf1bc912d4dd"
server_key_iv: "890fa95f979b16e63e83826fa7aef022"
server_key_version: "encryptionkey@0"
I20260926 23:29:14.422293 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/wal/instance:
uuid: "5f906dc8bec64e7eaba43a1777831684"
format_stamp: "Formatted at 2026-09-26 23:29:14 on dist-test-slave-q5g8"
server_key: "d79ae8eb98889a1220ddcf1bc912d4dd"
server_key_iv: "890fa95f979b16e63e83826fa7aef022"
server_key_version: "encryptionkey@0"
I20260926 23:29:14.428013 23159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:29:14.431789 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:14.433327 23159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:29:14.433538 23159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/wal
uuid: "5f906dc8bec64e7eaba43a1777831684"
format_stamp: "Formatted at 2026-09-26 23:29:14 on dist-test-slave-q5g8"
server_key: "d79ae8eb98889a1220ddcf1bc912d4dd"
server_key_iv: "890fa95f979b16e63e83826fa7aef022"
server_key_version: "encryptionkey@0"
I20260926 23:29:14.433774 23159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:14.461018 23159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:14.465967 23159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:14.466280 23159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:14.485633 23159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46177
I20260926 23:29:14.485695 23225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46177 every 8 connection(s)
I20260926 23:29:14.487100 23159 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:29:14.490939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 23159
I20260926 23:29:14.491132 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-0/wal/instance
I20260926 23:29:14.491264 23226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:14.491725 388 external_mini_cluster.cc:1428] Setting key fdb0c2c1b2a2b0380af7e531e338fef7
I20260926 23:29:14.501693 23226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:14 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 23:29:14.529531 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.505723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.97.62:46177)
0926 23:29:14.506633 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:14.506655 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:14.507109 (+ 454us) server_negotiation.cc:407] Connection header received
0926 23:29:14.509413 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:14.509420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:14.509707 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:14.509957 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:14.511436 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:14.513002 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:14.514758 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:14.515534 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:14.515825 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:14.521695 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:14.521705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:14.521720 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:14.521758 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:14.524825 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:14.525857 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:14.525865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:14.525867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:14.526053 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:14.526539 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:14.526542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:14.526544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:14.527083 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:14.527246 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:14.527980 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 23:29:14.528288 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":628,"thread_start_us":411,"threads_started":1}
I20260926 23:29:14.529587 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.505724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46177 (local address 127.0.0.1:48898)
0926 23:29:14.506931 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.506974 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:14.508979 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:14.509967 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:14.509974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:14.510807 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:14.511184 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:14.511205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:14.513201 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:14.513210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:14.514567 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:14.514584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:14.516121 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:14.516138 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:14.516481 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:14.517565 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:14.517623 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:14.521375 (+ 3752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:14.525008 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:14.525020 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:14.525035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:14.525681 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:14.526184 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:14.526192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:14.526198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:14.526426 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:14.527235 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:14.527246 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:14.527549 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 23:29:14.527851 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 23:29:14.528487 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":87,"thread_start_us":768,"threads_started":1}
I20260926 23:29:14.533032 23226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:29:14.534209 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.533677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51276)
0926 23:29:14.533891 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.534028 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 23:29:14.534508 23178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
I20260926 23:29:14.536170 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.535799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51286)
0926 23:29:14.535942 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.536023 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:29:14.536486 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 23:29:14.560225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43725
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46177,127.0.97.61:43725,127.0.97.60:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:29:14.585865 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } attempt: 1
I20260926 23:29:14.587123 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.586650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51292)
0926 23:29:14.586874 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.586984 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 23:29:14.588219 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.587849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51306)
0926 23:29:14.588002 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.588084 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:29:14.588441 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
I20260926 23:29:14.656792 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } attempt: 2
I20260926 23:29:14.658018 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.657506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51308)
0926 23:29:14.657783 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.657873 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 23:29:14.659075 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.658692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51318)
0926 23:29:14.658856 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.658947 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 23:29:14.659271 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
I20260926 23:29:14.750554 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } attempt: 3
I20260926 23:29:14.751760 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.751295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51320)
0926 23:29:14.751510 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.751615 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 23:29:14.752942 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.752527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51336)
0926 23:29:14.752718 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.752799 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 23:29:14.753217 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
I20260926 23:29:14.896517 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } attempt: 4
I20260926 23:29:14.897846 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.897258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51344)
0926 23:29:14.897613 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.897713 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 23:29:14.899008 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:14.898641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51350)
0926 23:29:14.898786 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:14.898871 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:29:14.899257 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43725: connect: Connection refused (error 111)
W20260926 23:29:14.918582 23233 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:29:14.918970 23233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:14.919100 23233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:14.919250 23233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:14.927100 23233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:14.927202 23233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:14.927302 23233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:14.927403 23233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:14.927506 23233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:14.938005 23233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46177,127.0.97.61:43725,127.0.97.60:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:14.939489 23233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:14.940941 23233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:14.953395 23238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:14.953687 23239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:14.953475 23241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:14.954689 23233 server_base.cc:1034] running on GCE node
I20260926 23:29:14.955545 23233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:14.957221 23233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:14.958498 23233 hybrid_clock.cc:648] HybridClock initialized: now 1790465354958421 us; error 37 us; skew 500 ppm
I20260926 23:29:14.958904 23233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:14 dist-test-slave-q5g8 krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:14.967891 23233 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 23:29:14.969956 23233 webserver.cc:460] Webserver started at http://127.0.97.61:41441/ using document root <none> and password file <none>
I20260926 23:29:14.970919 23233 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:14.971093 23233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:14.971474 23233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:14.975122 23233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data/instance:
uuid: "99bf5942e28245c48df181adddda89e0"
format_stamp: "Formatted at 2026-09-26 23:29:14 on dist-test-slave-q5g8"
server_key: "3186705a549b8e210e4ed6edc2920de7"
server_key_iv: "ee7479e4eee698cdfd62cd80414efec9"
server_key_version: "encryptionkey@0"
I20260926 23:29:14.976080 23233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/wal/instance:
uuid: "99bf5942e28245c48df181adddda89e0"
format_stamp: "Formatted at 2026-09-26 23:29:14 on dist-test-slave-q5g8"
server_key: "3186705a549b8e210e4ed6edc2920de7"
server_key_iv: "ee7479e4eee698cdfd62cd80414efec9"
server_key_version: "encryptionkey@0"
I20260926 23:29:14.982477 23233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:29:14.987053 23248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:14.988894 23233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:29:14.989154 23233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/wal
uuid: "99bf5942e28245c48df181adddda89e0"
format_stamp: "Formatted at 2026-09-26 23:29:14 on dist-test-slave-q5g8"
server_key: "3186705a549b8e210e4ed6edc2920de7"
server_key_iv: "ee7479e4eee698cdfd62cd80414efec9"
server_key_version: "encryptionkey@0"
I20260926 23:29:14.989437 23233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:15.009760 23233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:15.014580 23233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:15.014884 23233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:15.034627 23233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43725
I20260926 23:29:15.034667 23299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43725 every 8 connection(s)
I20260926 23:29:15.036140 23233 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:29:15.038125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 23233
I20260926 23:29:15.038344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-1/wal/instance
I20260926 23:29:15.038998 388 external_mini_cluster.cc:1428] Setting key 1bac5a707eb1a40b2464fcc7e8b827cd
I20260926 23:29:15.041522 23300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:15.051926 23300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:29:15.078369 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.054956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.0.97.62:46177)
0926 23:29:15.055503 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.055507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.056674 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 23:29:15.058875 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.058882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.058995 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.059137 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.061010 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.061781 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.063612 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.064089 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.064248 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.071355 (+ 7107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.071362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.071365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.071416 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.074508 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.075708 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.075713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.075715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.075848 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.076397 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.076400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.076402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.076771 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.076903 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.077882 (+ 979us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.078069 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":314,"threads_started":1}
I20260926 23:29:15.079551 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.055563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46177 (local address 127.0.0.1:48912)
0926 23:29:15.056476 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.056527 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.058542 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.059415 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.059425 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.060034 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.060676 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.060767 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.061979 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.061993 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.063434 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.063448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.064359 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.064367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.064650 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.065528 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.065556 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.071017 (+ 5461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.074707 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.074720 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.074745 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.075540 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.075975 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.075982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.075989 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.076277 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.076912 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.076924 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.077467 (+ 543us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.077794 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.078421 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":374,"threads_started":1}
I20260926 23:29:15.081656 23300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:29:15.098330 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.082374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51356)
0926 23:29:15.082538 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.082554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.082661 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.083667 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.083671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.084155 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.084290 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.084299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.085831 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.085839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.086739 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.086751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.087680 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.087686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.087885 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.088604 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.088630 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.091466 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.095531 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.095537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.095541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.095947 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.096381 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.096386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.096390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.096558 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.097296 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.097301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.097455 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.097602 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.097932 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 23:29:15.098464 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.082680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.0.97.61:43725)
0926 23:29:15.083238 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.083244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.083269 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:29:15.083313 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.083319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.083510 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.083672 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.084471 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.085650 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.086894 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.087513 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.087662 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.091778 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.091796 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.091814 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.091852 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.095351 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.096121 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.096130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.096132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.096261 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.096681 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.096689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.096721 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.097170 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.097398 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.097774 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.098081 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":408,"thread_start_us":269,"threads_started":1}
I20260926 23:29:15.101778 23300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:29:15.103147 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.102535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54506)
0926 23:29:15.102804 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.102982 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 23:29:15.103467 23251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 23:29:15.105217 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.104839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54522)
0926 23:29:15.104999 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.105080 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 23:29:15.105419 23300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.106338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
/tmp/dist-test-tasknM1jIl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38171
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46177,127.0.97.61:43725,127.0.97.60:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:29:15.154865 23300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 1
I20260926 23:29:15.156143 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.155675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54530)
0926 23:29:15.155869 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.155979 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:29:15.157279 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.156916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54536)
0926 23:29:15.157093 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.157166 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:29:15.157505 23300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.198544 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } attempt: 5
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:29:15.215764 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.199524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.0.97.61:43725)
0926 23:29:15.199730 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.199736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.199763 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:29:15.199982 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.199991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.200095 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.200247 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.201359 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.202465 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.203786 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.204453 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.204604 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.208368 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.208376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.208380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.208427 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.212266 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.213394 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.213401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.213404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.213687 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.214144 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.214150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.214154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.214544 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.214714 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.215137 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.215378 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:29:15.215763 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.199432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51364)
0926 23:29:15.199652 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.199678 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.199815 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.200323 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.200329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.200927 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.201148 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.201175 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.202639 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.202643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.203616 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.203630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.204646 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.204650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.204885 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.205628 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.205653 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.208117 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.212475 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.212480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.212484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.213209 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.213828 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.213833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.213837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.214000 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.214674 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.214679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.214853 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.215015 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.215373 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 23:29:15.217244 23226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:29:15.218663 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.218126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54542)
0926 23:29:15.218344 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.218451 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 23:29:15.219933 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.219525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54544)
0926 23:29:15.219700 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.219805 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 23:29:15.220189 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.225730 23300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 2
I20260926 23:29:15.226759 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.226332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54552)
0926 23:29:15.226511 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.226613 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 23:29:15.227838 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.227504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54564)
0926 23:29:15.227653 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.227731 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:29:15.228049 23300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.271495 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 1
I20260926 23:29:15.272986 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.272374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54572)
0926 23:29:15.272612 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.272777 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 23:29:15.274261 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.273838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54576)
0926 23:29:15.274008 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.274095 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 23:29:15.274665 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.319288 23300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 3
I20260926 23:29:15.320849 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.320212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54590)
0926 23:29:15.320474 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.320584 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 23:29:15.322242 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.321737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54592)
0926 23:29:15.322015 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.322106 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260926 23:29:15.322561 23300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.342941 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 2
I20260926 23:29:15.344226 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.343686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54594)
0926 23:29:15.343945 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.344044 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 23:29:15.345484 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.345116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54606)
0926 23:29:15.345281 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.345365 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 23:29:15.345719 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.451941 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 3
I20260926 23:29:15.453166 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.452643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54620)
0926 23:29:15.452890 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.453005 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:29:15.454226 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.453829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54630)
0926 23:29:15.454048 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.454120 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 23:29:15.454427 23226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
I20260926 23:29:15.465891 23300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 4
I20260926 23:29:15.467236 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.466706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54646)
0926 23:29:15.466958 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.467061 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 23:29:15.468402 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.468044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54654)
0926 23:29:15.468217 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.468288 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:29:15.468714 23300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38171: connect: Connection refused (error 111)
W20260926 23:29:15.468760 23308 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:29:15.469185 23308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:29:15.469308 23308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:29:15.469449 23308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:29:15.477360 23308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:29:15.477509 23308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:29:15.477636 23308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:29:15.477722 23308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:29:15.477834 23308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:29:15.488865 23308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46177,127.0.97.61:43725,127.0.97.60:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:29:15.490535 23308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:29:15.492586 23308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:29:15.506645 23313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:15.506671 23316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:29:15.506600 23314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:29:15.507336 23308 server_base.cc:1034] running on GCE node
I20260926 23:29:15.508466 23308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:29:15.510195 23308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:29:15.511451 23308 hybrid_clock.cc:648] HybridClock initialized: now 1790465355511358 us; error 65 us; skew 500 ppm
I20260926 23:29:15.511812 23308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:29:15.515434 23308 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 23:29:15.517412 23308 webserver.cc:460] Webserver started at http://127.0.97.60:44039/ using document root <none> and password file <none>
I20260926 23:29:15.518249 23308 fs_manager.cc:359] Metadata directory not provided
I20260926 23:29:15.518399 23308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:29:15.518777 23308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:29:15.522182 23308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data/instance:
uuid: "84d656c73f5f49168531cd4fe702e63f"
format_stamp: "Formatted at 2026-09-26 23:29:15 on dist-test-slave-q5g8"
server_key: "ba7001ac439f1ca7f01a07a931e40b52"
server_key_iv: "c981df783573b7d0f1029a452ce2e23f"
server_key_version: "encryptionkey@0"
I20260926 23:29:15.523077 23308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/wal/instance:
uuid: "84d656c73f5f49168531cd4fe702e63f"
format_stamp: "Formatted at 2026-09-26 23:29:15 on dist-test-slave-q5g8"
server_key: "ba7001ac439f1ca7f01a07a931e40b52"
server_key_iv: "c981df783573b7d0f1029a452ce2e23f"
server_key_version: "encryptionkey@0"
I20260926 23:29:15.528893 23308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:29:15.532740 23323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:29:15.534375 23308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:29:15.534615 23308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/wal
uuid: "84d656c73f5f49168531cd4fe702e63f"
format_stamp: "Formatted at 2026-09-26 23:29:15 on dist-test-slave-q5g8"
server_key: "ba7001ac439f1ca7f01a07a931e40b52"
server_key_iv: "c981df783573b7d0f1029a452ce2e23f"
server_key_version: "encryptionkey@0"
I20260926 23:29:15.534902 23308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:29:15.551115 23308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:29:15.555691 23308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:29:15.555996 23308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:29:15.577018 23308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38171
I20260926 23:29:15.577040 23374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38171 every 8 connection(s)
I20260926 23:29:15.578495 23308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:29:15.582643 23375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:29:15.584365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu as pid 23308
I20260926 23:29:15.584574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknM1jIl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790464978839002-388-0/minicluster-data/master-2/wal/instance
I20260926 23:29:15.585157 388 external_mini_cluster.cc:1428] Setting key 905a2b8669b5368dda302d831bce2178
I20260926 23:29:15.593560 23375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:29:15.612787 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.587112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.0.97.60:38171)
0926 23:29:15.588036 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.588050 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.588104 (+ 54us) server_negotiation.cc:407] Connection header received
0926 23:29:15.589877 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.589905 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.590402 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.590872 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.592771 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.594448 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.596521 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.597932 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.598141 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.603919 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.603933 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.603952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.604019 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.607852 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.608780 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.608791 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.608800 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.608978 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.609427 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.609430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.609432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.610020 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.610268 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.610914 (+ 646us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.611451 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":462,"threads_started":1}
I20260926 23:29:15.615357 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.597194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46177 (local address 127.0.0.1:48922)
0926 23:29:15.598130 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.598191 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.598369 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.598847 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.598853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.599850 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.600239 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.600258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.602060 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.602066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.603815 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.603831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.604815 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.604822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.605053 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.605723 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.605766 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.608636 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.612466 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.612477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.612485 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.613176 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.613592 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.613597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.613601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.613786 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.614383 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.614391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.614593 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.614769 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.615066 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":478,"threads_started":1}
I20260926 23:29:15.615882 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.597083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.0.97.62:46177)
0926 23:29:15.597702 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.597711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.598479 (+ 768us) server_negotiation.cc:407] Connection header received
0926 23:29:15.598551 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.598555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.598625 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.598724 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.600504 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.601869 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.604102 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.604551 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.604729 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.608917 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.608922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.608925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.608972 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.612310 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.613301 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.613305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.613308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.613455 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.613901 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.613904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.613906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.614209 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.614398 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.614947 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.615191 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":285,"threads_started":1}
I20260926 23:29:15.617463 23375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:29:15.631654 23226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 4
I20260926 23:29:15.633486 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.616559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.0.97.62:46177)
0926 23:29:15.616767 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.616773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.616790 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:29:15.616968 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.616972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.617042 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.617122 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.618581 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.619696 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.621361 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.622162 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.622318 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.626857 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.626864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.626869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.626918 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.630222 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.630986 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.630990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.630992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.631171 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.631658 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.631664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.631667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.632096 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.632459 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.632808 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.633071 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:29:15.633494 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.618314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.97.61:43725)
0926 23:29:15.618516 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.618523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.618554 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:29:15.618617 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.618622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.618683 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.618758 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.619674 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.620755 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.622507 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.623167 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.623306 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.627184 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.627189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.627192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.627230 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.630095 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.630893 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.630901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.630906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.631065 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.631521 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.631565 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.631572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.631889 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.632048 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.632853 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.633104 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:29:15.634230 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.618190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43725 (local address 127.0.0.1:51368)
0926 23:29:15.618364 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.618386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.618483 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.618809 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.618813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.619287 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.619470 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.619482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.620933 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.620937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.622332 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.622348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.623338 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.623343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.623518 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.624139 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.624158 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.626959 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.630268 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.630277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.630284 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.630747 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.631191 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.631197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.631202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.631398 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.632132 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.632137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.632335 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.632502 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.633864 (+ 1362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 23:29:15.635632 23375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:29:15.650516 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.632425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54684)
0926 23:29:15.632839 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.632862 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.632962 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.633903 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.633908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.634407 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.634635 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.634646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.636384 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.636392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.638403 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.638425 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.639658 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.639665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.639937 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.640667 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.640743 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.643523 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.647717 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.647723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.647728 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.648139 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.648580 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.648585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.648590 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.648829 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.649546 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.649553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.649742 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.649857 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.650151 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260926 23:29:15.651849 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.633272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.0.97.60:38171)
0926 23:29:15.633442 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.633449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.633471 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:29:15.633523 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.633528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.633605 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.633700 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.634851 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.636254 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.638600 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.639420 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.640332 (+ 912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.643770 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.643775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.643778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.643813 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.647573 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.648280 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.648286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.648291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.648437 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.648977 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.648982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.648985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.649369 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.651012 (+ 1643us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.651174 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.651435 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":70}
I20260926 23:29:15.656489 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.636512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54690)
0926 23:29:15.636880 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.636914 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.637032 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.639397 (+ 2365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.639446 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.640168 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.640366 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.640377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.642096 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.642105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.643579 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.643597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.645268 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.645275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.645536 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.646154 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.646182 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.648972 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.653447 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.653453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.653456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.653924 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.654363 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.654369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.654372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.654528 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.655235 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.655241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.655428 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.655602 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.655981 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260926 23:29:15.656589 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.635907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.97.61:43725)
0926 23:29:15.636102 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.636109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.636132 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:29:15.636191 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.636197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.636288 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.636395 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.638506 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.639767 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.641761 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.642374 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.642539 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.648555 (+ 6016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.648564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.648569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.648619 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.652578 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.653603 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.653609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.653612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.653786 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.654312 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.654323 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.654325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.654700 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.655020 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.655967 (+ 947us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.656249 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:29:15.657166 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.636577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.0.97.60:38171)
0926 23:29:15.639124 (+ 2547us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.639128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.639147 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:29:15.639184 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.639187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.639246 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.639383 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.640634 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.641902 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.643745 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.645061 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.645247 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.649252 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.649261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.649267 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.649317 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.653253 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.654070 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.654075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.654078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.654231 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.654666 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.654671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.654676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.655083 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.655258 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.655723 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.656109 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":2444,"thread_start_us":2187,"threads_started":1}
I20260926 23:29:15.659076 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684: Bootstrap starting.
I20260926 23:29:15.662787 23375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: Bootstrap starting.
I20260926 23:29:15.663077 23226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:15.664011 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:29:15.664359 23226 log.cc:824] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:15.666962 23375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:15.667778 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684: No bootstrap required, opened a new log
I20260926 23:29:15.668354 23375 log.cc:824] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:15.666885 23201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5f906dc8bec64e7eaba43a1777831684"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48936
I20260926 23:29:15.671813 23375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: No bootstrap required, opened a new log
I20260926 23:29:15.675211 23226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
I20260926 23:29:15.675680 23226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f906dc8bec64e7eaba43a1777831684, State: Initialized, Role: FOLLOWER
I20260926 23:29:15.676156 23226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
I20260926 23:29:15.677665 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.678128 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:15.678711 23226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:15.678850 23201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 23:29:15.679086 23201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:15.682964 23201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
I20260926 23:29:15.683676 23375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
I20260926 23:29:15.684173 23375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d656c73f5f49168531cd4fe702e63f, State: Initialized, Role: FOLLOWER
I20260926 23:29:15.684742 23375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
I20260926 23:29:15.685890 23390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.686229 23201 leader_election.cc:288] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [CANDIDATE]: Term 1 election: Requested vote from peers 99bf5942e28245c48df181adddda89e0 (127.0.97.61:43725), 84d656c73f5f49168531cd4fe702e63f (127.0.97.60:38171)
I20260926 23:29:15.686525 23390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:15.687470 23375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:15.687549 23350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f906dc8bec64e7eaba43a1777831684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "84d656c73f5f49168531cd4fe702e63f"
I20260926 23:29:15.688066 23350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:15.692710 23350 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f906dc8bec64e7eaba43a1777831684 in term 1.
I20260926 23:29:15.693784 23177 leader_election.cc:302] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f906dc8bec64e7eaba43a1777831684, 84d656c73f5f49168531cd4fe702e63f; no voters:
I20260926 23:29:15.694621 23384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:29:15.709093 23384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [term 1 LEADER]: Becoming Leader. State: Replica: 5f906dc8bec64e7eaba43a1777831684, State: Running, Role: LEADER
I20260926 23:29:15.711524 23384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
W20260926 23:29:15.719532 23180 tablet.cc:2369] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:29:15.730541 23397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f906dc8bec64e7eaba43a1777831684. Latest consensus state: current_term: 1 leader_uuid: "5f906dc8bec64e7eaba43a1777831684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.731887 23397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:15.737739 23409 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:29:15.752910 23411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:29:15.745656 23159 master_runner.cc:186] Error getting master registration for 127.0.97.61:43725: OK, instance_id { permanent_uuid: "99bf5942e28245c48df181adddda89e0" instance_seqno: 1790465355029143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:29:15.758042 23159 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:29:15.761170 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.734863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.0.97.60:38171)
0926 23:29:15.735144 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.735151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.737415 (+ 2264us) server_negotiation.cc:407] Connection header received
0926 23:29:15.739169 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.739176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.739275 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.739362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.743170 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.744281 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.746500 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.747209 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.747330 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.751516 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.751526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.751531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.751593 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.755941 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.757182 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.757189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.757193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.757387 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.758097 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.758104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.758111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.758722 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.759082 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.759985 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.760456 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:29:15.767987 23411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:29:15.767937 23300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } attempt: 5
Sep 26 23:29:15 dist-test-slave-q5g8 krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:29:15.785267 23418 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:29:15.789268 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.737386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.0.97.62:46177)
0926 23:29:15.737575 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.737584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.762030 (+ 24446us) server_negotiation.cc:407] Connection header received
0926 23:29:15.763359 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.763367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.763475 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.763597 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.766338 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.767483 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.770868 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.771648 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.771779 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.776833 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.776843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.776852 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.776915 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.784011 (+ 7096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.785015 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.785023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.785039 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.785249 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.785798 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.785806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.785809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.786239 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.786571 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.787789 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.788058 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 23:29:15.790392 23350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [term 1 FOLLOWER]: Refusing update from remote peer 5f906dc8bec64e7eaba43a1777831684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:29:15.791425 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:29:15.803114 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.770096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38171 (local address 127.0.0.1:54706)
0926 23:29:15.770314 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 23:29:15.770335 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:29:15.770450 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:29:15.771291 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:29:15.771301 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:29:15.772057 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:29:15.772361 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.772378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.774171 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.774180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.776090 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:29:15.776114 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.777798 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.777809 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:29:15.778115 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.778883 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:29:15.778914 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:29:15.785125 (+ 6211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:29:15.794377 (+ 9252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.794383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.794390 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.795055 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.796142 (+ 1087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:29:15.796149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:29:15.796153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:29:15.796641 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:29:15.798636 (+ 1995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:29:15.798642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:29:15.798994 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 23:29:15.799386 (+ 392us) client_negotiation.cc:240] Negotiation successful
0926 23:29:15.800655 (+ 1269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 23:29:15.805027 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.770499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.97.60:38171)
0926 23:29:15.770683 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.770690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.770712 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:29:15.770784 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.770793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.770893 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.771024 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.772625 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.773973 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.776318 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.777264 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.777453 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.785467 (+ 8014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.785475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.785480 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.785525 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.794126 (+ 8601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.795267 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.795273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.795278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.795761 (+ 483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.796969 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.796983 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.796986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.798406 (+ 1420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.800160 (+ 1754us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.801102 (+ 942us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.802604 (+ 1502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 23:29:15.808884 23329 tablet.cc:2369] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:29:15.795002 23308 master_runner.cc:186] Error getting master registration for 127.0.97.61:43725: OK, instance_id { permanent_uuid: "99bf5942e28245c48df181adddda89e0" instance_seqno: 1790465355029143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:29:15.814011 23300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0: Bootstrap starting.
I20260926 23:29:15.814333 23308 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:29:15.819072 23300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0: Neither blocks nor log segments found. Creating new log.
I20260926 23:29:15.821976 23300 log.cc:824] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:29:15.822196 23390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f906dc8bec64e7eaba43a1777831684. Latest consensus state: current_term: 1 leader_uuid: "5f906dc8bec64e7eaba43a1777831684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.822551 23390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:15.825013 23417 mvcc.cc:205] Tried to move back new op lower bound from 7333746097299546112 to 7333746097023995904. Current Snapshot: MvccSnapshot[applied={T|T < 7333746097299546112}]
I20260926 23:29:15.830968 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.762283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.0.97.61:43725)
0926 23:29:15.762555 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.762570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.765230 (+ 2660us) server_negotiation.cc:407] Connection header received
0926 23:29:15.765503 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.765513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.765627 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.765737 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.768174 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.769626 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.797365 (+ 27739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.798123 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.798245 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.802419 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:29:15.802427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:29:15.802433 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:29:15.802479 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:29:15.806842 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.820453 (+ 13611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.820461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.820465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.821090 (+ 625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:29:15.823242 (+ 2152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:29:15.823254 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:29:15.823261 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:29:15.824641 (+ 1380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:29:15.828332 (+ 3691us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.828884 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.829601 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 23:29:15.833505 23419 mvcc.cc:205] Tried to move back new op lower bound from 7333746097299546112 to 7333746097023995904. Current Snapshot: MvccSnapshot[applied={T|T < 7333746097299546112}]
I20260926 23:29:15.834630 23397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f906dc8bec64e7eaba43a1777831684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.834949 23397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:15.838474 23390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f906dc8bec64e7eaba43a1777831684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.838766 23300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0: No bootstrap required, opened a new log
I20260926 23:29:15.838793 23390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:15.841293 23411 catalog_manager.cc:1347] Generated new cluster ID: d7b0b8c72617444abb13b11186d20ff1
I20260926 23:29:15.841406 23411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:29:15.853060 23300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
I20260926 23:29:15.853574 23300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bf5942e28245c48df181adddda89e0, State: Initialized, Role: FOLLOWER
I20260926 23:29:15.854241 23300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } }
I20260926 23:29:15.859040 23411 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:29:15.860481 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.861088 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:15.862143 23300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:29:15.862663 23274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:29:15.867005 23274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [term 1 FOLLOWER]: Refusing update from remote peer 5f906dc8bec64e7eaba43a1777831684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:29:15.867847 23275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f906dc8bec64e7eaba43a1777831684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99bf5942e28245c48df181adddda89e0"
I20260926 23:29:15.868109 23397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:29:15.872182 23275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f906dc8bec64e7eaba43a1777831684 in term 1.
I20260926 23:29:15.874155 23411 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:29:15.895020 23411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684: Generated new TSK 0
I20260926 23:29:15.896226 23411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:29:15.899943 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f906dc8bec64e7eaba43a1777831684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.900432 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:15.905269 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f906dc8bec64e7eaba43a1777831684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.905385 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f906dc8bec64e7eaba43a1777831684. Latest consensus state: current_term: 1 leader_uuid: "5f906dc8bec64e7eaba43a1777831684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f906dc8bec64e7eaba43a1777831684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46177 } } peers { permanent_uuid: "99bf5942e28245c48df181adddda89e0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43725 } } peers { permanent_uuid: "84d656c73f5f49168531cd4fe702e63f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38171 } } }
I20260926 23:29:15.905640 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f906dc8bec64e7eaba43a1777831684 [sys.catalog]: This master's current role is: LEADER
I20260926 23:29:15.905685 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:29:15.910346 23435 catalog_manager.cc:1259] Loaded cluster ID: d7b0b8c72617444abb13b11186d20ff1
I20260926 23:29:15.910481 23435 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0: loading cluster ID for follower catalog manager: success
I20260926 23:29:15.914158 23435 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0: acquiring CA information for follower catalog manager: success
I20260926 23:29:15.916965 23435 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99bf5942e28245c48df181adddda89e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 23:29:15.975198 23379 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:15.970056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.0.97.62:46177)
0926 23:29:15.970301 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.970309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.973005 (+ 2696us) server_negotiation.cc:407] Connection header received
0926 23:29:15.973330 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.973344 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.973443 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.973534 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.974867 (+ 1333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48948: BlockingRecv error: recv got EOF from 127.0.0.1:48948 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 23:29:15.976488 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:15.970540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.0.97.60:38171)
0926 23:29:15.970789 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.970800 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.973005 (+ 2205us) server_negotiation.cc:407] Connection header received
0926 23:29:15.974466 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.974473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.974588 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.974671 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.976159 (+ 1488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54708: BlockingRecv error: recv got EOF from 127.0.0.1:54708 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 23:29:15.978710 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:15.973592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.0.97.61:43725)
0926 23:29:15.973786 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.973793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.976175 (+ 2382us) server_negotiation.cc:407] Connection header received
0926 23:29:15.976435 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.976442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.976565 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.976650 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:15.978373 (+ 1723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51394: BlockingRecv error: recv got EOF from 127.0.0.1:51394 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:29:16.000241 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:15.985207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.0.97.61:43725)
0926 23:29:15.985453 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:15.985463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:15.993763 (+ 8300us) server_negotiation.cc:407] Connection header received
0926 23:29:15.994094 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:15.994101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:15.994450 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:15.994675 (+ 225us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:15.995215 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.996055 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.997634 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:15.998124 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:15.998293 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:15.998870 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:15.999461 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:15.999650 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:15.999715 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 23:29:15.999813 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 23:29:16.010876 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.007459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.0.97.60:38171)
0926 23:29:16.007697 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.007702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.009256 (+ 1554us) server_negotiation.cc:407] Connection header received
0926 23:29:16.009554 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.009559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.009671 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.009803 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.010679 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54724: BlockingRecv error: recv got EOF from 127.0.0.1:54724 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:29:16.023125 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.016576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.0.97.61:43725)
0926 23:29:16.016913 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.016923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.017503 (+ 580us) server_negotiation.cc:407] Connection header received
0926 23:29:16.017762 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.017770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.017857 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.017971 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.018497 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.019300 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.020783 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.021464 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.021629 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.022241 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.022390 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.022509 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.022639 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.022727 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
W20260926 23:29:16.033521 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.030758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.0.97.60:38171)
0926 23:29:16.031020 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.031031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.031685 (+ 654us) server_negotiation.cc:407] Connection header received
0926 23:29:16.031986 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.032003 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.032123 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.032205 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.033305 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54734: BlockingRecv error: recv got EOF from 127.0.0.1:54734 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 23:29:16.087009 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.079316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.0.97.61:43725)
0926 23:29:16.079508 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.079514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.081361 (+ 1847us) server_negotiation.cc:407] Connection header received
0926 23:29:16.081626 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.081632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.081737 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.081850 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.082395 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.083243 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.085009 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.085507 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.085678 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.086108 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.086262 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.086370 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.086532 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.086632 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 23:29:16.096375 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.094144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.0.97.60:38171)
0926 23:29:16.094344 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.094350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.094880 (+ 530us) server_negotiation.cc:407] Connection header received
0926 23:29:16.095110 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.095117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.095213 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.095326 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.096115 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54748: BlockingRecv error: recv got EOF from 127.0.0.1:54748 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:29:16.116968 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.110865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.0.97.61:43725)
0926 23:29:16.111219 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.111229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.111858 (+ 629us) server_negotiation.cc:407] Connection header received
0926 23:29:16.112142 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.112149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.112239 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.112352 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.112828 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.113629 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.115037 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.115550 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.115702 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.116100 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.116267 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.116376 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.116530 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.116618 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
W20260926 23:29:16.125952 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.124134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.0.97.60:38171)
0926 23:29:16.124432 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.124440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.124463 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:29:16.124749 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.124755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.124848 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.124971 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.125758 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54754: BlockingRecv error: recv got EOF from 127.0.0.1:54754 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 23:29:16.144016 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.138061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.0.97.61:43725)
0926 23:29:16.138279 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.138290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.138891 (+ 601us) server_negotiation.cc:407] Connection header received
0926 23:29:16.139181 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.139187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.139314 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.139404 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.139819 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.140751 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.142131 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.142620 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.142758 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.143176 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.143319 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.143420 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.143573 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.143662 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 23:29:16.152669 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.150299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.0.97.60:38171)
0926 23:29:16.150541 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.150547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.151105 (+ 558us) server_negotiation.cc:407] Connection header received
0926 23:29:16.151374 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.151380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.151470 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.151589 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.152393 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54770: BlockingRecv error: recv got EOF from 127.0.0.1:54770 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:29:16.179834 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.172088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.0.97.61:43725)
0926 23:29:16.172257 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.172262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.174510 (+ 2248us) server_negotiation.cc:407] Connection header received
0926 23:29:16.174755 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.174760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.174836 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.174936 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.175374 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.176220 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.177897 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.178356 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.178500 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.178948 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.179148 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.179262 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.179407 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.179495 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 23:29:16.188657 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.186407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.0.97.60:38171)
0926 23:29:16.186675 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.186685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.187215 (+ 530us) server_negotiation.cc:407] Connection header received
0926 23:29:16.187420 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.187425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.187515 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.187628 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.188415 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54780: BlockingRecv error: recv got EOF from 127.0.0.1:54780 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:29:16.230165 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.224336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.0.97.61:43725)
0926 23:29:16.224621 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.224631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.225243 (+ 612us) server_negotiation.cc:407] Connection header received
0926 23:29:16.225480 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.225485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.225598 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.225704 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.226163 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.226907 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.228248 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.228806 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.228947 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.229349 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.229455 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.229553 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.229715 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.229801 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 23:29:16.238581 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.236158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.0.97.60:38171)
0926 23:29:16.236408 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.236418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.236997 (+ 579us) server_negotiation.cc:407] Connection header received
0926 23:29:16.237213 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.237220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.237386 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.237505 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.238330 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54788: BlockingRecv error: recv got EOF from 127.0.0.1:54788 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:29:16.320032 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.312608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.0.97.61:43725)
0926 23:29:16.312838 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.312845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.313439 (+ 594us) server_negotiation.cc:407] Connection header received
0926 23:29:16.313700 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.313707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.313811 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.313931 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.314466 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.315681 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.317715 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.318379 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.318493 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.319128 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.319294 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.319360 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.319626 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.319715 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 23:29:16.333451 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.331216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.0.97.60:38171)
0926 23:29:16.331420 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.331427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.331954 (+ 527us) server_negotiation.cc:407] Connection header received
0926 23:29:16.332154 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.332161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.332252 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.332356 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.333225 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54790: BlockingRecv error: recv got EOF from 127.0.0.1:54790 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 23:29:16.470003 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.464941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.0.97.61:43725)
0926 23:29:16.465114 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.465119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.465492 (+ 373us) server_negotiation.cc:407] Connection header received
0926 23:29:16.465660 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.465664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.465736 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.465821 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.466185 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.466897 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.468133 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.468556 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.468844 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.469264 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.469369 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.469465 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.469623 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.469707 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 23:29:16.478276 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.476305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.97.60:38171)
0926 23:29:16.476558 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.476566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.476828 (+ 262us) server_negotiation.cc:407] Connection header received
0926 23:29:16.477130 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.477134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.477213 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.477301 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.478061 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54806: BlockingRecv error: recv got EOF from 127.0.0.1:54806 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:29:16.743495 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:16.738175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.0.97.61:43725)
0926 23:29:16.738375 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.738384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.738778 (+ 394us) server_negotiation.cc:407] Connection header received
0926 23:29:16.739017 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.739022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.739094 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.739198 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:16.739650 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.740395 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.741771 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:16.742232 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:16.742379 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:16.742778 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:16.742880 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:16.742985 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:16.743139 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 23:29:16.743220 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 23:29:16.750907 23381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:29:16.748987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.0.97.60:38171)
0926 23:29:16.749211 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:16.749219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:16.749395 (+ 176us) server_negotiation.cc:407] Connection header received
0926 23:29:16.749675 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:16.749679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:16.749767 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:16.749872 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:29:16.750643 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54820: BlockingRecv error: recv got EOF from 127.0.0.1:54820 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:29:16.789965 23418 catalog_manager.cc:1259] Loaded cluster ID: d7b0b8c72617444abb13b11186d20ff1
I20260926 23:29:16.790084 23418 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: loading cluster ID for follower catalog manager: success
I20260926 23:29:16.792613 23418 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: acquiring CA information for follower catalog manager: success
I20260926 23:29:16.794657 23418 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84d656c73f5f49168531cd4fe702e63f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:29:17.275641 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:17.270167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.0.97.61:43725)
0926 23:29:17.270997 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:17.271001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:17.271020 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:29:17.271114 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:17.271118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:17.271185 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:17.271359 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:17.271849 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.272713 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.274054 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.274434 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.274523 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:17.274870 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:17.274964 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:17.275029 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:17.275210 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 23:29:17.275273 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":622,"threads_started":1}
I20260926 23:29:17.288393 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:17.282166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.0.97.60:38171)
0926 23:29:17.283258 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:17.283262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:17.283278 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:29:17.283336 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:17.283340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:17.283493 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:17.283591 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:17.283975 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.284652 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.286147 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.286538 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.286628 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:17.287135 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:17.287781 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:17.287933 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:17.287996 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 23:29:17.288067 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":886,"threads_started":1}
I20260926 23:29:17.302802 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:17.294912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.0.97.60:38171)
0926 23:29:17.295101 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:17.295109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:17.296155 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 23:29:17.296472 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:17.296481 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:17.296585 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:17.296660 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:17.297720 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.298761 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.300904 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.301340 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.301492 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:17.301990 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:17.302087 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:17.302155 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:17.302344 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 23:29:17.302439 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:29:17.304239 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:17.294729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.0.97.62:46177)
0926 23:29:17.295538 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:17.295548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:17.295901 (+ 353us) server_negotiation.cc:407] Connection header received
0926 23:29:17.296256 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:17.296264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:17.296547 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:17.296745 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:17.297493 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.298657 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.301654 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.302302 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.302436 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:17.303027 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:17.303517 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:17.303643 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:17.303759 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 23:29:17.303849 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":496,"threads_started":1}
I20260926 23:29:17.304634 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:29:17.295978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.0.97.61:43725)
0926 23:29:17.296193 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:29:17.296200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:29:17.296911 (+ 711us) server_negotiation.cc:407] Connection header received
0926 23:29:17.297211 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:29:17.297218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:29:17.297310 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:29:17.297426 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:29:17.298092 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.298815 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.302581 (+ 3766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:29:17.303254 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:29:17.303401 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:29:17.303850 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:29:17.303987 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:29:17.304052 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 23:29:17.304282 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 23:29:17.304374 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:29:17.309580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 23159
I20260926 23:29:17.310564 23221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:29:17.419334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 23159
W20260926 23:29:17.439563 23324 connection.cc:535] server connection from 127.0.0.1:54684 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:29:17.439965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 23233
I20260926 23:29:17.440975 23295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:29:17.549475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 23233
I20260926 23:29:17.569646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu pid 23308
I20260926 23:29:17.570773 23370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:29:17.679003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknM1jIl/build/asan/bin/kudu with pid 23308
2026-09-26T23:29:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5102 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9923 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (378794 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST